1

I newly joined an organisation and we recently introduced a Data Warehouse solution (Snowflake) that incorporates a large amount of external systems (CRM etc). There are use cases to bring manual data input on weekly (e.i. Sales targets ). This one area that I am having trouble with.

In an ideal world, all systems would perfectly integrate and form the core data within the DW.

But the reality is that there is likely to need to keep the manual data input to create a complete picture (at least until we can find a way around it long term).

So far I have thought of Excel/Google Sheet as manual entry into a backend service which populates DB Tables in the staging server.

Does anyone here have experience in this scenario? How do users of a data platform typically handle this scenario? And practice for handling manual data entry into a Data Warehouse solution?

Any help you can provide here would be greatly appreciated.

Nick.Mc
  • 18,304
  • 6
  • 61
  • 91
timy
  • 97
  • 6
  • Yes it's completely normal to require additional reference data, budgets etc. The solution depends on your budget and user requirements. The lowest tech solution is to have people manage excel files and have these imported automatically every day but this is very error prone given that the slightest amount of rubbish entered will mess up the process. You'd need to work out what import formats are supported by Snowflake, and work out a mechanism to feed back errors to end users. The next option is to actually build a low tech application to allow users to enter validated data into a table – Nick.Mc Nov 22 '21 at 00:28
  • I've removed "azure sql data warehouse" as that doesn't appear to be in your technology mix. If you are using Azure, the best low-code solution to use is PowerApps. You can get a weba app writing into an Azure SQL database really quickly, then have a batch process load this into snowflake. – Nick.Mc Nov 22 '21 at 00:30

0 Answers0