I am trying to build GCP Cloud Functions which are triggered via Cloud Scheduler to collect odds data from bookmakers API's for a variety of different sports and competitions that run on different schedules.
I am looking for some advice on my approach about if it is the right strategy or if there is a better way to achieve what I want.
I have been using python for the code and GCP Cloud Functions (Gen 2), Cloud Scheduler, Pub/Sub. Firestore.
Objective: To use collect odds data from bookmakers API feeds and store desired data points from the feeds in Firestore.
Plan: Setup Cloud Functions (Gen 2) with Cloud Scheduler to collect the odds data for each sports competition which runs on a range of different schedules.
Problem: Each sport will have 3-5 betting markets collected for each event. Sometimes the bookmaker API feed requires that you make one API call to get event Ids, then another call using those event ids in the API URL to get the events odds data. So there are 3-5 API calls to collect all the required betting market data for each game.
At the moment I have those different API calls set up as different functions and it seems to work all ok when I run it locally.
Approach: I would like to set up each of those functions on Cloud Functions (Gen 2) using a One to Many PubSub Messaging System.
- Function 1 (event id collection - sends event ids to pubsub topic)
- Function 2 (odds market 1 collection using event ids from pubsub message of which this function is a subscriber)
- Function 3 (odds market 2 collection using event ids from pubsub message of which this function is a subscriber)
I would need function 1 to run first (cloud scheduler), collect the data and push to pubsub, then once that message is in pubsub I would need functions 1 and 2 to run once the message is in pubsub and pulled to function.
This is the code for Function 1 which collected the event id's for the games:
import requests
import json
from firebase_admin import firestore
from google.cloud import pubsub_v1
db = firestore.Client(project='xxxxxxx')
# API INFO
Base_url = 'https://xxxxxxx.net/v1/feeds/sportsbookv2'
Sport_id = '1000093204'
AppID = 'xxxxxxx'
AppKey = 'xxxxxxx'
Country = 'en_AU'
Site = 'xxxxxxx'
project_id = "xxxxxxx"
topic_id = "xxxxxxx_basketball_nba"
publisher = pubsub_v1.PublisherClient()
# The `topic_path` method creates a fully qualified identifier
# in the form `projects/{project_id}/topics/{topic_id}`
topic_path = publisher.topic_path(project_id, topic_id)
def win_odds(request):
event_ids = []
url = f"{Base_url}/betoffer/event/{','.join(map(str, event_ids))}.json?app_id={AppID}&app_key={AppKey}&local={Country}&site={Site}"
print(url)
windata = requests.get(url).text
windata = json.loads(windata)
for odds_data in windata['betOffers']:
if odds_data['betOfferType']['name'] == 'Head to Head' and 'MAIN' in odds_data['tags']:
event_id = odds_data['eventId']
home_team = odds_data['outcomes'][0]['participant']
home_team_win_odds = odds_data['outcomes'][0]['odds']
away_team = odds_data['outcomes'][1]['participant']
away_team_win_odds = odds_data['outcomes'][1]['odds']
print(f'{event_id} {home_team} {home_team_win_odds} {away_team} {away_team_win_odds}')
event_ids.append(event_id)
# WRITE TO FIRESTORE
doc_ref = db.collection(u'xxxxxxx_au').document(u'basketball_nba').collection(u'win_odds').document(
f'{event_id}')
doc_ref.set({
u'event_id': event_id,
u'home_team': home_team,
u'home_team_win_odds': home_team_win_odds,
u'away_team': away_team,
u'away_team_win_odds': away_team_win_odds,
u'timestamp': firestore.SERVER_TIMESTAMP,
})
return event_ids
print(f"Published messages to {topic_path}.")
Below is the code for the functions that will pull the event ids from pubsub and use in their API calls to collect the data on each betting market.
Below is the template for the subscriber functions which will pull the event ids message from pubsub subscription and then use those event ids in the API call for that function.
from concurrent.futures import TimeoutError
from google.cloud import pubsub_v1
import requests
import json
from firebase_admin import firestore
import functions_framework
db = firestore.Client(project='xxxxxxx')
# API INFO
Base_url = 'https://xxxxxxx.net/v1/feeds/sportsbookv2'
Sport_id = '1000093204'
AppID = 'xxxxxxx'
AppKey = 'xxxxxxx'
Country = 'en_AU'
Site = 'xxxxxxx'
project_id = "xxxxxxx"
subscription_id = "xxxxxxx_basketball_nba_events"
timeout = 5.0
subscriber = pubsub_v1.SubscriberClient()
subscription_path = subscriber.subscription_path(project_id, subscription_id)
@functions_framework.cloud_event
def win_odds(message):
print(message.data)
event_ids = json.loads(message.data)
url = f"{Base_url}/betoffer/event/{','.join(map(str, event_ids))}.json?app_id={AppID}&app_key={AppKey}&local={Country}&site={Site}"
print(url)
windata = requests.get(url).text
windata = json.loads(windata)
for odds_data in windata['betOffers']:
if odds_data['betOfferType']['name'] == 'Head to Head' and 'MAIN' in odds_data['tags']:
event_id = odds_data['eventId']
home_team = odds_data['outcomes'][0]['participant']
home_team_win_odds = odds_data['outcomes'][0]['odds']
away_team = odds_data['outcomes'][1]['participant']
away_team_win_odds = odds_data['outcomes'][1]['odds']
print(f'{event_id} {home_team} {home_team_win_odds} {away_team} {away_team_win_odds}')
# WRITE TO FIRESTORE
doc_ref = db.collection(u'xxxxxxx_au').document(u'basketball_nba').collection(u'win_odds').document(
f'{event_id}')
doc_ref.set({
u'event_id': event_id,
u'home_team': home_team,
u'home_team_win_odds': home_team_win_odds,
u'away_team': away_team,
u'away_team_win_odds': away_team_win_odds,
u'timestamp': firestore.SERVER_TIMESTAMP,
})
if __name__ == '__main__':
streaming_pull_future = subscriber.subscribe(subscription_path, callback=win_odds)
print(f"Listening for messages on {subscription_path}..\n")
# Wrap subscriber in a 'with' block to automatically call close() when done.
with subscriber:
try:
# When `timeout` is not set, result() will block indefinitely,
# unless an exception is encountered first.
streaming_pull_future.result(timeout=timeout)
except TimeoutError:
streaming_pull_future.cancel() # Trigger the shutdown.
streaming_pull_future.result() # Block until the shutdown is complete.
print("Listening, then shutting down.")
