Real-Time & Historical Job Stats
Easily monitor jobs in real-time, compare jobs to previous jobs run, spot trends forming and catch problems early before your data feeds go down.
Item & Page Vaildation
ScrapeOps checks pages for CAPTCHAs & bans, and the data quality of every item scraped so you can detect broken parsers without having to query your DB.
Job Scheduling & Management
Plan, schedule, manage and run all your jobs across multiple servers with a single, easy to use scraper management dashboard. Integrates via Scrapyd and SSH.
Automatic Health Checks
ScrapeOps automatically checks every job versus historical averages and custom health checks, then tells you if your scrapers are healthy or not.
Custom Checks, Alerts & Reports
Easily configure your own custom health checks and reports, and get alerts how and when you want them. Alerts via Slack, email and more.
Proxy Analytics
Monitor your current proxy usage & performance, and get alerts when you are running out of credits or there is a problem.
Real-Time Job Monitoring
Easily Monitor Your Scrapers
ScrapeOps automatically logs and ships your scraping performance stats, so you can monitor everything in a single dashboard.
Effortlessly compare pages & items scraped, runtimes, status codes, success rates and errors versus previous job runs to identify potential issues with your scrapers.
Health Checks & Alerts
Built-In Health Checks & Alerts
Out of the box, ScrapeOps automatically checks the health of every job versus its historical moving average to see if it is healthy or not. Alerting you when and how you want it.
If that isn't enough, you can configure custom checks, alerts and reports on any job or spider.
Integration
Integrate With Scrapy in 30 Seconds
Web scraping can be painful enough without having to build your own custom monitoring and alerting system from scratch.
With just 3 lines of code, ScrapeOps gives you all the monitoring and alerting options you will ever need straight out of the box.
pip install scrapeops-scrapy
## settings.py
# Add Your ScrapeOps API key
SCRAPEOPS_API_KEY = 'YOUR_API_KEY'
# Add In The ScrapeOps Extension
EXTENSIONS = {
'scrapeops_scrapy.extension.ScrapeOpsMonitor': 500,
}
# Update The Download Middlewares
DOWNLOADER_MIDDLEWARES = {
'scrapeops_scrapy.middleware.retry.RetryMiddleware': 550,
'scrapy.downloadermiddlewares.retry.RetryMiddleware': None,
}
## DONE!
More Integrations On The Way
Python Requests
Python's most popular HTTP request library.
Python Selenium
Python's most popular headless browser.
NodeJs
NodeJs popular Requests, Axios, fetch libraries.
Puppeteer
NodeJs most popular headless browser library.
Get Started For Free
If you have a hobby project or need only basic monitoring then use our Community Plan for free, forever!.
Unlock Premium Features
If web scraped data is mission critical for your business, then ScrapeOps premium features will make your life a whole lot easier.
Startup Plan
Startup Plan
Business Plan
Business Plan
Need a bigger plan? Contact us here.
Premium Features Explained
Real-Time Monitoring & Alerts
ScrapeOps will monitor your jobs in real-time, and alert you if their performance drops instead of only at job finish.
Data Quality Validation
Monitor your items field coverage and setup custom unit tests so you can detect parser issues straight away.
Captcha/Ban Detection & Retries
ScrapeOps will check every scraped page for CAPTCHAs & bans and trigger an event so you can retry the request.
Custom Health Checks & Alerts
Create custom real-time scraper health checks for all your scrapers so you detect unhealthy jobs straight away. Simple to setup with Slack, email or phone.
Custom Periodic Reports
Automate your daily scraping checks by scheduling ScrapeOps to check your spiders & jobs every couple hours and send you a report if any issues are detected.
Custom Scraper Events & Tags
Trigger custom events in your scrapers to log any invalid HTML responses, geotargeting or data quality issues that are specific to your own scrapers.
Scheduling via SSH
Give ScrapeOps SSH access to your servers, and you will be able to schedule and run any type of scraper from the dashboard. No more setting up CRON jobs.
ETL Pipeline Support
Monitor scraping and data parsing across multiple servers even when you have split HTML extraction and data parsing between different processes.
Distributed Crawl Support
Monitor your scrapers even when you have a distributed scraping infrastructure, where multiple scraping processes are working from the same queue.