0

So I am currently implementing my very first Django-based web app. However, I figured that I only need Django to perform a backend cron job to scrape data from a website and then update existing data in a PostgreSQL database. Then I just use a React frontend to retrieve data from the database and visualize it on the webpage.

My issue now is that I don't know how to conceptually tackle this challenge. Currently, I have a model in my models.py file that successfully created my empty table in PostgreSQL:

from django.db import models

# Create your models here.
class rainAreas(models.Model):
    Country = models.CharField(max_length=100)
    HasRain= models.BooleanField()
    Since = models.DateField()

    class Meta:
        app_label = "rain_areas"

I also filled the table manually with dummy data. Finally, I have a script in my admin.py file, which successfully creates the desired list of data scraped from a website. It looks like this:

my_data = [{"country": "Germany", "HasRain": True, "Since": "2020-08-11"}, {"country": "France",....

But now I am stuck. What is the next move to

  1. make an SQL UPDATE on the table I created with the data I have in admin.py
  2. transform this script to a cron job which should run every hour
JoeBe
  • 1,224
  • 3
  • 13
  • 28

1 Answers1

1

there are basically two options:

  1. try to use celery
  2. write a custom admin comand

Adding celery might add to much complexity.

if you go for the second option you have to ensure to use the correct python environment in your script/ in the cronjob check this answer

Updating the entries would be first retrieving and then saving the new value

for d in my_data:
    rainArea = RainArea.objects.get(country=d['country'])
    rainArea.has_rain = d['HasRain']
    rainArea.since =  datetime.strptime(d['Since'], '%y-%m-%d')
    rainArea.save()

Please note that i slightly refactored the wording to better follow pep8:

  • in python classes start uppercase
  • django models are singular named and not plural (RainArea)
  • attributes usually don't start with uppercase
bb4L
  • 899
  • 5
  • 16