Essential Infrastructure Decisions for Stable Automated Scraping
When we talk about running spiders and automated scripts, we typically mean that the code would be deployed in a cloud-based server. Among the most typically utilized and inexpensive solutions is AWS-EC2 by Amazon. It assists you run code on a Linux or a Windows server which is handled and maintained by their team at AWS.
You are charged just for the uptime and you can stop your server in case you prepare not to use it for some time. Establishing your scraping infrastructure on the cloud can prove to be very inexpensive and efficient in the long run, however you will need cloud designers to set things up and look after upgrading them or making modifications to them as and when needed.
In case you are scraping high-res data such as images or videos which run into GBs, you can attempt AWS-S3, which is the most affordable data-storage solution on the market today. There are more pricey options that you can select depending upon how regularly you wish to access the data. In case you are extracting specific data-points, you can store the data in a database such as Postgres in AWS-RDS.
When scraping a single web page, you can run the script from your laptop and finish the job. But in case you are trying to crawl data from thousands of web-pages of a single site every second, you will be blacklisted and blocked from the website in less than minutes.
