0

I am interested in the comments (=text) made on certain YouTube Channels. I have scraped data with the Google YouTube Data API. The data comes in a complex structure and format (see picture below) that I am trying to disentangle for a research project.

The comments are stored in the fields Text Display and Text Original that belong to the dictionary Snippet, which in turn is part of the dictionary Top Level Comments. Top Level Comments is part of a string list that in turn is part of the dictionary items.

I think I need to subset the dictionary Top Level Comment as all the comments and related information (see picture below) I need are stored in nested dictionaries there. I don't think I can access the dictionary Top Level Comment as it is part of the list Snippet. So I first tried to subset the list Snippet. This is where I am stuck.

Here my code so far:

from googleapiclient.discovery import build
api_key = '_______________________________'

youtube = build('youtube', 'v3', developerKey = api_key)

#find channel ID https://commentpicker.com/youtube-channel-id.php
request = youtube.commentThreads().list(
    part = 'snippet',
    allThreadsRelatedToChannelId = 'UC_zxivooFdvF4uuBosUnJxQ'
    )

response3 = request.execute()

##Code to explore data structure and format is excluded 

#subset dictionary according to keys we want 
includedKeys = ['items']
dataDic = {k:v for k, v in response3.items() if k in includedKeys}

In below code I unsuccessfully tried to subset the list Snipet in different ways or convert it.

dataDic2 = {x['snippet'] for x in dataDic} #Link no 1
#TypeError: string indices must be integers
dataDic2 = {x['snippet'] for x in dataDic} #Link no 1
#TypeError: string indices must be integers

dataDic2 = [{'snippet': d['snippet']} for d in dataDic] #Link no 2
#TypeError: string indices must be integers 

dataDic2 = [topLevelComment['snippet'] for topLevelComment in dataDic['topLevelComment']['snippet']] #Link no 3
#KeyError: 'topLevelComment'import ast

result = ast.literal_eval('[snippet]') 
assert type(result) is list #Link no 4 and 5
#ValueError: malformed node or string: <_ast.Name object at 0x0000010F6D7B9A08>

Link no 1 Link no 2 Link no 3
Link no 4 Link no 5
This link says that ast.literal.eval does not work with lists and dictionaries?

So finally - how to retrieve the data? I need all fields circled in red in the picture showing the data structure. data structure

EDIT: sample data

Simone
  • 497
  • 5
  • 19
  • share the ACTUAL data you get from you tube and the desired output (As a **text**) – balderman Sep 15 '21 at 11:36
  • @balderman - Your comment is standard.... I described the date including a picture of the data structure. But if you need the data for a response. Sure I can share data if you like to strain your eyes through all the {} and []..... – Simone Sep 15 '21 at 11:42
  • Yes please - add it to the post (you can add a *subset* only) but make sure those are valid data structures – balderman Sep 15 '21 at 11:44
  • @balderman Data up-loaded. Looking VERY forward to your ideas and solutions. :-) – Simone Sep 15 '21 at 11:50
  • I can see the input but not the requested output. Can you add this as well? What is the field that holds the comment? is it `textDisplay` ? – balderman Sep 15 '21 at 11:52
  • @balderman: output was already described --> all the fields that are circled in red in the picture. Really, it can be in a pandas DF, numpy DF, dictionary just NOT in a nested structure so I can further process it. – Simone Sep 15 '21 at 12:32

1 Answers1

0

see below

data = {'kind': 'youtube#commentThreadListResponse', 'etag': '_yOZ67ear9btS5RarXfH3Xir6A8',
        'nextPageToken': 'QURTSl9pME5DS2FQZm5yRzZ5b0ZGZUJGeENkMGh2UWxzVjNueEdUVmtmbVVqYksxSmN4QnpBdDFFWkpCREl6REZVQmlHZS1makpfZXFkQzFNbEpwbDFpb0dNWm95Z2E1TE03NE5GWEg0ajE5UWt0bnlpYS1PczlFVWZ1a1hqbTJLREVRempJaVpaRTYtcnpFeUM2ZU5Va1hUSHR5cVJFTEJ2akdtOHFkTWhGdmdmWUZsMUMwUHg0eTZNVzFBZVdsd1A0YXBqaWhnNGVNMXc=',
        'pageInfo': {'totalResults': 14, 'resultsPerPage': 20}, 'items': [
        {'kind': 'youtube#commentThread', 'etag': 'knxvgtYnhlPIpkevoCXSTZamb40', 'id': 'Ugwmdd9KdDm4Hm7MxlJ4AaABAg',
         'snippet': {'channelId': 'UC_zxivooFdvF4uuBosUnJxQ', 'videoId': 'tUXWw6WvgkI',
                     'topLevelComment': {'kind': 'youtube#comment', 'etag': '4m76jMeR8qFmfrk42kfKeA5Iv_Y',
                                         'id': 'Ugwmdd9KdDm4Hm7MxlJ4AaABAg',
                                         'snippet': {'channelId': 'UC_zxivooFdvF4uuBosUnJxQ', 'videoId': 'tUXWw6WvgkI',
                                                     'textDisplay': 'Tipp 1: Zusatzszüge – machs wie Fredy. (Hinweis: dieses Video wurde vor der Corona-Pandemie erstellt)',
                                                     'textOriginal': 'Tipp 1: Zusatzszüge – machs wie Fredy. (Hinweis: dieses Video wurde vor der Corona-Pandemie erstellt)',
                                                     'authorDisplayName': 'Zuschauerquaeler',
                                                     'authorProfileImageUrl': 'https://yt3.ggpht.com/ytc/AKedOLQCWIoN-3MmDfxflS5ipDVvatDw8TpbD43mn2kb=s48-c-k-c0x00ffffff-no-rj',
                                                     'authorChannelUrl': 'http://www.youtube.com/channel/UCECxysNsTQLhrelU2KikMjQ',
                                                     'authorChannelId': {'value': 'UCECxysNsTQLhrelU2KikMjQ'},
                                                     'canRate': True, 'viewerRating': 'none', 'likeCount': 1,
                                                     'publishedAt': '2021-09-15T07:29:00Z',
                                                     'updatedAt': '2021-09-15T07:29:00Z'}}, 'canReply': True,
                     'totalReplyCount': 0, 'isPublic': True}},
        {'kind': 'youtube#commentThread', 'etag': 'tq7mSQltdzKz0sthUiAIPYrQgJg', 'id': 'Ugy2jzL0838zj9HyHu94AaABAg',
         'snippet': {'channelId': 'UC_zxivooFdvF4uuBosUnJxQ', 'videoId': 'M98TRem03Lg',
                     'topLevelComment': {'kind': 'youtube#comment', 'etag': '8BDnS6DXuaN8VdFzHsj7dc1YPZc',
                                         'id': 'Ugy2jzL0838zj9HyHu94AaABAg',
                                         'snippet': {'channelId': 'UC_zxivooFdvF4uuBosUnJxQ', 'videoId': 'M98TRem03Lg',
                                                     'textDisplay': 'Ich sehe das Kulturland schon schmelzen und verschwinden...',
                                                     'textOriginal': 'Ich sehe das Kulturland schon schmelzen und verschwinden...',
                                                     'authorDisplayName': 'Janik Von Niederhäusern',
                                                     'authorProfileImageUrl': 'https://yt3.ggpht.com/ytc/AKedOLSk69KdiWMSYw0sYQSBdjEHagXJTD9tWlHdsw=s48-c-k-c0x00ffffff-no-rj',
                                                     'authorChannelUrl': 'http://www.youtube.com/channel/UCt87CYDxeIbDRRJLVT0VrdQ',
                                                     'authorChannelId': {'value': 'UCt87CYDxeIbDRRJLVT0VrdQ'},
                                                     'canRate': True, 'viewerRating': 'none', 'likeCount': 0,
                                                     'publishedAt': '2021-09-14T18:08:55Z',
                                                     'updatedAt': '2021-09-14T18:08:55Z'}}, 'canReply': True,
                     'totalReplyCount': 1, 'isPublic': True}},
        {'kind': 'youtube#commentThread', 'etag': 'h_gpfnmUju60NWNxlFEwxjkIPQU', 'id': 'Ugx5GfaJTwt5cnuQ3Bh4AaABAg',
         'snippet': {'channelId': 'UC_zxivooFdvF4uuBosUnJxQ', 'videoId': 'M98TRem03Lg',
                     'topLevelComment': {'kind': 'youtube#comment', 'etag': 'fMmN1zDH7PVIWbw3L0n5Mt0dtqk',
                                         'id': 'Ugx5GfaJTwt5cnuQ3Bh4AaABAg',
                                         'snippet': {'channelId': 'UC_zxivooFdvF4uuBosUnJxQ', 'videoId': 'M98TRem03Lg',
                                                     'textDisplay': 'Guete initiativ! Mega fan vo dere projekt!',
                                                     'textOriginal': 'Guete initiativ! Mega fan vo dere projekt!',
                                                     'authorDisplayName': 'Nionity',
                                                     'authorProfileImageUrl': 'https://yt3.ggpht.com/ytc/AKedOLTM-Tj3pWLuyhuH7ivlUwxs4YtQn6gez-BMCLdLzQ=s48-c-k-c0x00ffffff-no-rj',
                                                     'authorChannelUrl': 'http://www.youtube.com/channel/UCbUj9ZwI0YOkElVEfpAnBVQ',
                                                     'authorChannelId': {'value': 'UCbUj9ZwI0YOkElVEfpAnBVQ'},
                                                     'canRate': True, 'viewerRating': 'none', 'likeCount': 0,
                                                     'publishedAt': '2021-09-14T07:18:31Z',
                                                     'updatedAt': '2021-09-14T07:18:31Z'}}, 'canReply': True,
                     'totalReplyCount': 0, 'isPublic': True}},
        {'kind': 'youtube#commentThread', 'etag': 'LOajqt43iY4A2N4V0yiLBRZwaig', 'id': 'Ugxez_tcF7ts7VaAL7t4AaABAg',
         'snippet': {'channelId': 'UC_zxivooFdvF4uuBosUnJxQ', 'videoId': 'zYnbgDyWM9o',
                     'topLevelComment': {'kind': 'youtube#comment', 'etag': 'DvNHOkNftBCLBqV1Ajam8mzMFYg',
                                         'id': 'Ugxez_tcF7ts7VaAL7t4AaABAg',
                                         'snippet': {'channelId': 'UC_zxivooFdvF4uuBosUnJxQ', 'videoId': 'zYnbgDyWM9o',
                                                     'textDisplay': 'Très mauvaise voix off, à un moment il y se reprend même dans le texte ',
                                                     'textOriginal': 'Très mauvaise voix off, à un moment il y se reprend même dans le texte ',
                                                     'authorDisplayName': 'Patrick__EPfan',
                                                     'authorProfileImageUrl': 'https://yt3.ggpht.com/ytc/AKedOLTOmUsxVCimwNSQBVPxNUXfFbUNuYnN7VzVEeBUJA=s48-c-k-c0x00ffffff-no-rj',
                                                     'authorChannelUrl': 'http://www.youtube.com/channel/UC8DxMAk8T9Gv8RW0f2n0Q2w',
                                                     'authorChannelId': {'value': 'UC8DxMAk8T9Gv8RW0f2n0Q2w'},
                                                     'canRate': True, 'viewerRating': 'none', 'likeCount': 0,
                                                     'publishedAt': '2021-09-12T12:12:58Z',
                                                     'updatedAt': '2021-09-12T12:12:58Z'}}, 'canReply': True,
                     'totalReplyCount': 0, 'isPublic': True}},
        {'kind': 'youtube#commentThread', 'etag': 'MGsQS-TUcYHnuyjyN932wpVIM_A', 'id': 'UgxYTxqSwAsyGyOHzU94AaABAg',
         'snippet': {'channelId': 'UC_zxivooFdvF4uuBosUnJxQ', 'videoId': '4nU0MgKft6c',
                     'topLevelComment': {'kind': 'youtube#comment', 'etag': 'iRkZfQGVCGFZ13s8D3xrVZQw83A',
                                         'id': 'UgxYTxqSwAsyGyOHzU94AaABAg',
                                         'snippet': {'channelId': 'UC_zxivooFdvF4uuBosUnJxQ', 'videoId': '4nU0MgKft6c',
                                                     'textDisplay': 'Shiey be like', 'textOriginal': 'Shiey be like',
                                                     'authorDisplayName': 'Canopener Guy',
                                                     'authorProfileImageUrl': 'https://yt3.ggpht.com/2XG9uyYmOfkeubUNFQR0cgj7xCimKLsg6_r-3E1PTPVLixXjcxeFosF1HoytvHibGJrxQXal=s48-c-k-c0x00ffffff-no-rj',
                                                     'authorChannelUrl': 'http://www.youtube.com/channel/UCk8pieRaYyzsnU32Gp85DvA',
                                                     'authorChannelId': {'value': 'UCk8pieRaYyzsnU32Gp85DvA'},
                                                     'canRate': True, 'viewerRating': 'none', 'likeCount': 0,
                                                     'publishedAt': '2021-09-02T23:23:35Z',
                                                     'updatedAt': '2021-09-02T23:23:35Z'}}, 'canReply': True,
                     'totalReplyCount': 0, 'isPublic': True}},
        {'kind': 'youtube#commentThread', 'etag': 'bcPCCsMbvquhAKLiEqIR4a20HnA', 'id': 'Ugw8FWvl7Hbf1RvJWhV4AaABAg',
         'snippet': {'channelId': 'UC_zxivooFdvF4uuBosUnJxQ', 'videoId': 'oxSLp_1WtcM',
                     'topLevelComment': {'kind': 'youtube#comment', 'etag': 'rTl4oSjvH14OF4xQ1mnM_amfZag',
                                         'id': 'Ugw8FWvl7Hbf1RvJWhV4AaABAg',
                                         'snippet': {'channelId': 'UC_zxivooFdvF4uuBosUnJxQ', 'videoId': 'oxSLp_1WtcM',
                                                     'textDisplay': 'Vivement un Lyria en Belgique !!!!',
                                                     'textOriginal': 'Vivement un Lyria en Belgique !!!!',
                                                     'authorDisplayName': 'Kayuchi Fujimoto',
                                                     'authorProfileImageUrl': 'https://yt3.ggpht.com/ytc/AKedOLQ9YSDYj2tQFvjKjt9F_CH9AR2dcWrr84jA70am=s48-c-k-c0x00ffffff-no-rj',
                                                     'authorChannelUrl': 'http://www.youtube.com/channel/UCe5ctUAG-Z7cU_hpc-CbauQ',
                                                     'authorChannelId': {'value': 'UCe5ctUAG-Z7cU_hpc-CbauQ'},
                                                     'canRate': True, 'viewerRating': 'none', 'likeCount': 0,
                                                     'publishedAt': '2021-09-02T21:39:26Z',
                                                     'updatedAt': '2021-09-02T21:39:26Z'}}, 'canReply': True,
                     'totalReplyCount': 0, 'isPublic': True}},
        {'kind': 'youtube#commentThread', 'etag': 'qbrUI9Z2YkM3LtYOqFogVRwcZWE', 'id': 'UgwomjMWUx5CHjlU_ox4AaABAg',
         'snippet': {'channelId': 'UC_zxivooFdvF4uuBosUnJxQ', 'videoId': '8vCvSmAIv1s',
                     'topLevelComment': {'kind': 'youtube#comment', 'etag': 'gYjvyBgNsZUB_FYUDK20LCVU-Qk',
                                         'id': 'UgwomjMWUx5CHjlU_ox4AaABAg',
                                         'snippet': {'channelId': 'UC_zxivooFdvF4uuBosUnJxQ', 'videoId': '8vCvSmAIv1s',
                                                     'textDisplay': 'Build a high speed railway line into the moon I dare you with 20 million francs',
                                                     'textOriginal': 'Build a high speed railway line into the moon I dare you with 20 million francs',
                                                     'authorDisplayName': 'Simulated Trainspotter',
                                                     'authorProfileImageUrl': 'https://yt3.ggpht.com/3P-cR_3ORURRZH5RYImCeFv0yeC64SHtpS3otsCiGn4AuBXG-tQVrqnG32vJm4bfwxRt3MwCDzw=s48-c-k-c0x00ffffff-no-rj',
                                                     'authorChannelUrl': 'http://www.youtube.com/channel/UCF4ganYY8qP9q8YwXpDn2tQ',
                                                     'authorChannelId': {'value': 'UCF4ganYY8qP9q8YwXpDn2tQ'},
                                                     'canRate': True, 'viewerRating': 'none', 'likeCount': 0,
                                                     'publishedAt': '2021-09-02T08:36:45Z',
                                                     'updatedAt': '2021-09-02T08:36:45Z'}}, 'canReply': True,
                     'totalReplyCount': 0, 'isPublic': True}},
        {'kind': 'youtube#commentThread', 'etag': '5KVenAu6Nn6RdnpKTpPj49KuYRY', 'id': 'UgyXleqDMoHFnid0OpV4AaABAg',
         'snippet': {'channelId': 'UC_zxivooFdvF4uuBosUnJxQ', 'videoId': '7earPWDJbhA',
                     'topLevelComment': {'kind': 'youtube#comment', 'etag': 'C3AxUnPxhDZuIYAKsjqeIZxmyQI',
                                         'id': 'UgyXleqDMoHFnid0OpV4AaABAg',
                                         'snippet': {'channelId': 'UC_zxivooFdvF4uuBosUnJxQ', 'videoId': '7earPWDJbhA',
                                                     'textDisplay': 'Sehr schön', 'textOriginal': 'Sehr schön',
                                                     'authorDisplayName': 'Pranave4 Roblox',
                                                     'authorProfileImageUrl': 'https://yt3.ggpht.com/V_qXZAr4xsbi2GEFJ2t8NhwDYWGEeiBhFCgVYcgs1TwmaS1e6gCwktKZpdNPJszs3Zwu71ZZ2w=s48-c-k-c0x00ffffff-no-rj',
                                                     'authorChannelUrl': 'http://www.youtube.com/channel/UCKoDZxOJY6e90jeujtkC_4A',
                                                     'authorChannelId': {'value': 'UCKoDZxOJY6e90jeujtkC_4A'},
                                                     'canRate': True, 'viewerRating': 'none', 'likeCount': 2,
                                                     'publishedAt': '2021-08-27T16:06:59Z',
                                                     'updatedAt': '2021-08-27T16:06:59Z'}}, 'canReply': True,
                     'totalReplyCount': 0, 'isPublic': True}},
        {'kind': 'youtube#commentThread', 'etag': 'mH33Uu3Bm3zkVGLZDiOaOg2idSM', 'id': 'UgxQRQaVxnzeFQRTPTp4AaABAg',
         'snippet': {'channelId': 'UC_zxivooFdvF4uuBosUnJxQ', 'videoId': 'AXMw3vtsswY',
                     'topLevelComment': {'kind': 'youtube#comment', 'etag': 'Sht8Gm_LShDQ9cKfIl1nH53FgsI',
                                         'id': 'UgxQRQaVxnzeFQRTPTp4AaABAg',
                                         'snippet': {'channelId': 'UC_zxivooFdvF4uuBosUnJxQ', 'videoId': 'AXMw3vtsswY',
                                                     'textDisplay': 'wie kann mann feuerwehr mann bei SBB werden',
                                                     'textOriginal': 'wie kann mann feuerwehr mann bei SBB werden',
                                                     'authorDisplayName': 'Florian Ruhland',
                                                     'authorProfileImageUrl': 'https://yt3.ggpht.com/ytc/AKedOLQNfiz21ybCpfDmaXKefJtuy1UDHwFenhsL0R14Kg=s48-c-k-c0x00ffffff-no-rj',
                                                     'authorChannelUrl': 'http://www.youtube.com/channel/UCS7LfiWU_ebI-E3ny8Yb6PA',
                                                     'authorChannelId': {'value': 'UCS7LfiWU_ebI-E3ny8Yb6PA'},
                                                     'canRate': True, 'viewerRating': 'none', 'likeCount': 0,
                                                     'publishedAt': '2021-08-21T11:00:05Z',
                                                     'updatedAt': '2021-08-21T11:00:05Z'}}, 'canReply': True,
                     'totalReplyCount': 1, 'isPublic': True}},
        {'kind': 'youtube#commentThread', 'etag': 'oM57z1ZCosWjFXPDl1VMIQIFpJ8', 'id': 'UgzzHV3cayZFI7MpziB4AaABAg',
         'snippet': {'channelId': 'UC_zxivooFdvF4uuBosUnJxQ', 'videoId': 'DmBo0MMxDb0',
                     'topLevelComment': {'kind': 'youtube#comment', 'etag': '-ecKB_iUT-BOVOeNfX7qoAr0poI',
                                         'id': 'UgzzHV3cayZFI7MpziB4AaABAg',
                                         'snippet': {'channelId': 'UC_zxivooFdvF4uuBosUnJxQ', 'videoId': 'DmBo0MMxDb0',
                                                     'textDisplay': 'I am only 15, but i have a very very big passion for these trains, i can’t wait to drive around Switzerland and help people arrive at their destinations<br>I also learned about signals in Switzerland as short documentaries on how these trains work.<br>I hope nothing major will change in 5 years:) i really dreaming of becoming an engine driver',
                                                     'textOriginal': 'I am only 15, but i have a very very big passion for these trains, i can’t wait to drive around Switzerland and help people arrive at their destinations\nI also learned about signals in Switzerland as short documentaries on how these trains work.\nI hope nothing major will change in 5 years:) i really dreaming of becoming an engine driver',
                                                     'authorDisplayName': 'Fred Dev',
                                                     'authorProfileImageUrl': 'https://yt3.ggpht.com/JEaQIjszQdpIDgsrIKEtIX6KaeryO48U4IcbSl45oFIKrDNoCxwhmWh3fC6exW5X1pL15Hiw4w=s48-c-k-c0x00ffffff-no-rj',
                                                     'authorChannelUrl': 'http://www.youtube.com/channel/UCJKarhI8HsHHix0-HckXwVg',
                                                     'authorChannelId': {'value': 'UCJKarhI8HsHHix0-HckXwVg'},
                                                     'canRate': True, 'viewerRating': 'none', 'likeCount': 1,
                                                     'publishedAt': '2021-08-19T22:32:58Z',
                                                     'updatedAt': '2021-08-19T22:32:58Z'}}, 'canReply': True,
                     'totalReplyCount': 1, 'isPublic': True}},
        {'kind': 'youtube#commentThread', 'etag': 'Xu5rUasdLD7ZFsRPWPrL2JUJCWg', 'id': 'UgwBkkcOhrjuzFjE6Y54AaABAg',
         'snippet': {'channelId': 'UC_zxivooFdvF4uuBosUnJxQ', 'videoId': 'ES0AnIBNJfQ',
                     'topLevelComment': {'kind': 'youtube#comment', 'etag': '1ps-PTcq7S2TzbY7s4OuafI4-Fg',
                                         'id': 'UgwBkkcOhrjuzFjE6Y54AaABAg',
                                         'snippet': {'channelId': 'UC_zxivooFdvF4uuBosUnJxQ', 'videoId': 'ES0AnIBNJfQ',
                                                     'textDisplay': 'wie heisst der sprecher dieser werbung? so eine wunderbare stimme!<br>die musik ist auch toll, wie heisst das stück?',
                                                     'textOriginal': 'wie heisst der sprecher dieser werbung? so eine wunderbare stimme!\ndie musik ist auch toll, wie heisst das stück?',
                                                     'authorDisplayName': 'cloudwalker',
                                                     'authorProfileImageUrl': 'https://yt3.ggpht.com/ytc/AKedOLQxGBcardOjutARwZxXcfbUSH3f66gqTzq3EA=s48-c-k-c0x00ffffff-no-rj',
                                                     'authorChannelUrl': 'http://www.youtube.com/channel/UC3VmTS8W5GKZf0PeIb8l2Jw',
                                                     'authorChannelId': {'value': 'UC3VmTS8W5GKZf0PeIb8l2Jw'},
                                                     'canRate': True, 'viewerRating': 'none', 'likeCount': 1,
                                                     'publishedAt': '2021-08-18T00:50:32Z',
                                                     'updatedAt': '2021-08-18T00:50:32Z'}}, 'canReply': True,
                     'totalReplyCount': 2, 'isPublic': True}},
        {'kind': 'youtube#commentThread', 'etag': '_hlBnClge81P8_RqsXR7q4_BIes', 'id': 'Ugzvldq2VB0lBIzoGVR4AaABAg',
         'snippet': {'channelId': 'UC_zxivooFdvF4uuBosUnJxQ', 'videoId': 'AXMw3vtsswY',
                     'topLevelComment': {'kind': 'youtube#comment', 'etag': 'QZFjHr5bIQC72OicksbfJ3Py-Hk',
                                         'id': 'Ugzvldq2VB0lBIzoGVR4AaABAg',
                                         'snippet': {'channelId': 'UC_zxivooFdvF4uuBosUnJxQ', 'videoId': 'AXMw3vtsswY',
                                                     'textDisplay': 'Ihr seid spitze! Danke, dass es euch gibt ',
                                                     'textOriginal': 'Ihr seid spitze! Danke, dass es euch gibt ',
                                                     'authorDisplayName': 'Cris Tiano',
                                                     'authorProfileImageUrl': 'https://yt3.ggpht.com/ytc/AKedOLT_ZmzCfLD22VLmHv-zIOnNiBGZHoYBhgcsgQ=s48-c-k-c0x00ffffff-no-rj',
                                                     'authorChannelUrl': 'http://www.youtube.com/channel/UCU3xXx609PrAf6AwLjs5oSw',
                                                     'authorChannelId': {'value': 'UCU3xXx609PrAf6AwLjs5oSw'},
                                                     'canRate': True, 'viewerRating': 'none', 'likeCount': 0,
                                                     'publishedAt': '2021-08-16T15:53:30Z',
                                                     'updatedAt': '2021-08-16T15:53:30Z'}}, 'canReply': True,
                     'totalReplyCount': 0, 'isPublic': True}},

]}

comments = []
for item in data['items']:
    entry = {}
    snippet = item['snippet']['topLevelComment']['snippet']
    for field in ['channelId', 'videoId']:
        entry[field] = snippet[field]
    for field in ['textOriginal', 'textDisplay','canRate','likeCount','updatedAt','viewerRating','publishedAt']:
        entry[field] = snippet[field]
    entry['canReply'] = item['snippet']['canReply']
    entry['isPublic'] = item['snippet']['isPublic']
    entry['totalReplyCount'] = item['snippet']['totalReplyCount']
    comments.append(entry)
for idx,comment in enumerate(comments,1):
    print(f'{idx}) {comment}')

output

1) {'channelId': 'UC_zxivooFdvF4uuBosUnJxQ', 'videoId': 'tUXWw6WvgkI', 'textOriginal': 'Tipp 1: Zusatzszüge – machs wie Fredy. (Hinweis: dieses Video wurde vor der Corona-Pandemie erstellt)', 'textDisplay': 'Tipp 1: Zusatzszüge – machs wie Fredy. (Hinweis: dieses Video wurde vor der Corona-Pandemie erstellt)', 'canRate': True, 'likeCount': 1, 'updatedAt': '2021-09-15T07:29:00Z', 'viewerRating': 'none', 'publishedAt': '2021-09-15T07:29:00Z', 'canReply': True, 'isPublic': True, 'totalReplyCount': 0}
2) {'channelId': 'UC_zxivooFdvF4uuBosUnJxQ', 'videoId': 'M98TRem03Lg', 'textOriginal': 'Ich sehe das Kulturland schon schmelzen und verschwinden...', 'textDisplay': 'Ich sehe das Kulturland schon schmelzen und verschwinden...', 'canRate': True, 'likeCount': 0, 'updatedAt': '2021-09-14T18:08:55Z', 'viewerRating': 'none', 'publishedAt': '2021-09-14T18:08:55Z', 'canReply': True, 'isPublic': True, 'totalReplyCount': 1}
3) {'channelId': 'UC_zxivooFdvF4uuBosUnJxQ', 'videoId': 'M98TRem03Lg', 'textOriginal': 'Guete initiativ! Mega fan vo dere projekt!', 'textDisplay': 'Guete initiativ! Mega fan vo dere projekt!', 'canRate': True, 'likeCount': 0, 'updatedAt': '2021-09-14T07:18:31Z', 'viewerRating': 'none', 'publishedAt': '2021-09-14T07:18:31Z', 'canReply': True, 'isPublic': True, 'totalReplyCount': 0}
4) {'channelId': 'UC_zxivooFdvF4uuBosUnJxQ', 'videoId': 'zYnbgDyWM9o', 'textOriginal': 'Très mauvaise voix off, à un moment il y se reprend même dans le texte ', 'textDisplay': 'Très mauvaise voix off, à un moment il y se reprend même dans le texte ', 'canRate': True, 'likeCount': 0, 'updatedAt': '2021-09-12T12:12:58Z', 'viewerRating': 'none', 'publishedAt': '2021-09-12T12:12:58Z', 'canReply': True, 'isPublic': True, 'totalReplyCount': 0}
5) {'channelId': 'UC_zxivooFdvF4uuBosUnJxQ', 'videoId': '4nU0MgKft6c', 'textOriginal': 'Shiey be like', 'textDisplay': 'Shiey be like', 'canRate': True, 'likeCount': 0, 'updatedAt': '2021-09-02T23:23:35Z', 'viewerRating': 'none', 'publishedAt': '2021-09-02T23:23:35Z', 'canReply': True, 'isPublic': True, 'totalReplyCount': 0}
6) {'channelId': 'UC_zxivooFdvF4uuBosUnJxQ', 'videoId': 'oxSLp_1WtcM', 'textOriginal': 'Vivement un Lyria en Belgique !!!!', 'textDisplay': 'Vivement un Lyria en Belgique !!!!', 'canRate': True, 'likeCount': 0, 'updatedAt': '2021-09-02T21:39:26Z', 'viewerRating': 'none', 'publishedAt': '2021-09-02T21:39:26Z', 'canReply': True, 'isPublic': True, 'totalReplyCount': 0}
7) {'channelId': 'UC_zxivooFdvF4uuBosUnJxQ', 'videoId': '8vCvSmAIv1s', 'textOriginal': 'Build a high speed railway line into the moon I dare you with 20 million francs', 'textDisplay': 'Build a high speed railway line into the moon I dare you with 20 million francs', 'canRate': True, 'likeCount': 0, 'updatedAt': '2021-09-02T08:36:45Z', 'viewerRating': 'none', 'publishedAt': '2021-09-02T08:36:45Z', 'canReply': True, 'isPublic': True, 'totalReplyCount': 0}
8) {'channelId': 'UC_zxivooFdvF4uuBosUnJxQ', 'videoId': '7earPWDJbhA', 'textOriginal': 'Sehr schön', 'textDisplay': 'Sehr schön', 'canRate': True, 'likeCount': 2, 'updatedAt': '2021-08-27T16:06:59Z', 'viewerRating': 'none', 'publishedAt': '2021-08-27T16:06:59Z', 'canReply': True, 'isPublic': True, 'totalReplyCount': 0}
9) {'channelId': 'UC_zxivooFdvF4uuBosUnJxQ', 'videoId': 'AXMw3vtsswY', 'textOriginal': 'wie kann mann feuerwehr mann bei SBB werden', 'textDisplay': 'wie kann mann feuerwehr mann bei SBB werden', 'canRate': True, 'likeCount': 0, 'updatedAt': '2021-08-21T11:00:05Z', 'viewerRating': 'none', 'publishedAt': '2021-08-21T11:00:05Z', 'canReply': True, 'isPublic': True, 'totalReplyCount': 1}
10) {'channelId': 'UC_zxivooFdvF4uuBosUnJxQ', 'videoId': 'DmBo0MMxDb0', 'textOriginal': 'I am only 15, but i have a very very big passion for these trains, i can’t wait to drive around Switzerland and help people arrive at their destinations\nI also learned about signals in Switzerland as short documentaries on how these trains work.\nI hope nothing major will change in 5 years:) i really dreaming of becoming an engine driver', 'textDisplay': 'I am only 15, but i have a very very big passion for these trains, i can’t wait to drive around Switzerland and help people arrive at their destinations<br>I also learned about signals in Switzerland as short documentaries on how these trains work.<br>I hope nothing major will change in 5 years:) i really dreaming of becoming an engine driver', 'canRate': True, 'likeCount': 1, 'updatedAt': '2021-08-19T22:32:58Z', 'viewerRating': 'none', 'publishedAt': '2021-08-19T22:32:58Z', 'canReply': True, 'isPublic': True, 'totalReplyCount': 1}
11) {'channelId': 'UC_zxivooFdvF4uuBosUnJxQ', 'videoId': 'ES0AnIBNJfQ', 'textOriginal': 'wie heisst der sprecher dieser werbung? so eine wunderbare stimme!\ndie musik ist auch toll, wie heisst das stück?', 'textDisplay': 'wie heisst der sprecher dieser werbung? so eine wunderbare stimme!<br>die musik ist auch toll, wie heisst das stück?', 'canRate': True, 'likeCount': 1, 'updatedAt': '2021-08-18T00:50:32Z', 'viewerRating': 'none', 'publishedAt': '2021-08-18T00:50:32Z', 'canReply': True, 'isPublic': True, 'totalReplyCount': 2}
12) {'channelId': 'UC_zxivooFdvF4uuBosUnJxQ', 'videoId': 'AXMw3vtsswY', 'textOriginal': 'Ihr seid spitze! Danke, dass es euch gibt ', 'textDisplay': 'Ihr seid spitze! Danke, dass es euch gibt ', 'canRate': True, 'likeCount': 0, 'updatedAt': '2021-08-16T15:53:30Z', 'viewerRating': 'none', 'publishedAt': '2021-08-16T15:53:30Z', 'canReply': True, 'isPublic': True, 'totalReplyCount': 0}
balderman
  • 22,927
  • 7
  • 34
  • 52
  • This only solves part of the problem. I need all the fields circled in red. – Simone Sep 15 '21 at 12:37
  • Don't you think you can take it from here...? – balderman Sep 15 '21 at 14:43
  • Code works. Thanks. Lists of dictionaries was a bit difficult to convert into a DF as none of the frequently suggested code for conversion worked. Always had NaN for channelId and video Id introduced. So spend some good time googling around. (I usually use R, not Python) Finally `dataDF = pd.DataFrame.from_dict(comments, orient='columns') ` did the trick --> I like to provide people with complete solutions as I am grateful to complete solutions with good descriptions as to why things happen too. – Simone Sep 17 '21 at 08:42