Website Data Monitoring

Applies to: Pro Plan Enterprise Plan

Add Screenshots provides businesses with the ability to monitor and track website changes over time, allowing them to stay ahead of competitors and ensure that their website is functioning properly. By converting HTML to JSON and storing and monitoring the data for changes, businesses can easily track changes, perform audits, and conduct automated testing, leading to improved decision-making and increased customer satisfaction.

Use Cases for Monitoring and Tracking Website Changes

There are many use cases for tracking website changes using HTML to JSON conversion and monitoring, such as:

Keep track and watch out for price changes or discounts on a product you're interested in buying
Stay on top of your competitors by monitoring their website and stay informed when they make a change
Stay in touch with your customers by getting notified whenever new reviews are posted about your business or service online
Monitor product pages for items that are frequently out of stock and get notified when they become available
Monitor websites of favorite writers, filmmakers, or musicians and be notified when new content is released
Book appointments by monitoring popular websites and being the first one to know whenever a slot opens up

How to Monitor a Website for Changes

There are two ways to evaluate a webpage for changes:

Scheduled: Define an interval to regularly monitor a website (e.g., hourly, daily, weekly).
On-demand: Send a web request to check for changes immediately (e.g., convert HTML to JSON and monitor data).

Comparing Scheduled with Track Changes Data

Scheduled data monitoring is ideal when you'd like to capture and store data at regular intervals. On the other hand, track changes data monitoring is suitable when you'd like to capture and store changed data at regular intervals and receive notifications when a change is detected.

Using the approach above to track website changes, it is possible to achieve:

Receive notifications with the current and previous data when a change is detected.
Determine when and how often a competitor updates their data.
Track what has changed at a particular time.
Show the latest data using a fixed file path.

For example, capture website data of your competitor's pricing page may produce the following files:

pricing 2020-08-01.json
pricing 2020-08-02.json
...
pricing 2020-08-31.json

Scheduled data changes are great when you'd like to capture and store changed data at a regular interval and receive an email notification when a change was detected. For example, capturing your competitor's pricing on a daily basis may produce the following files:

pricing.json (last version)
pricing 2020-08-07.json (send email notification)
pricing 2020-08-25.png (send email notification)

Using the approach above to track slow changing website data, it is possible to achieve the following:

Receive an email notification with the current and previous data when a change was detected
Determine when and how often a competitor updates their prices (e.g. twice this month)
Save on storage by only keeping JSON data files when a change was detected
Show the data of the latest prices using a fixed file path (pricing.json)

A comparison between scheduled and track data changes:

Description	Scheduled	Track Changes
Run at scheduled intervals
Only store changed data
Track when data was changed
Track what data changed
Send an email notification when a change was detected

Track website changes on demand

The following request properties are available on the data object in the payload:

Parameter	Required	Description
file	Required	The name or pattern (without the extension) of the JSON file. Daily Example: use `{filepath}_{date}` to save the date when the change was detected. Time Example: use `{filepath}_{datetime}` to save the date and time when the change was detected.
folder	Optional	The folder name or pattern to store JSON files to Example: use `track-changes` to save the JSON file in the "track-changes" folder.
pretty	Optional	Set to true for pretty print formatting of JSON data
image	Optional	Set to true to add a screenshot image
track_changes	Optional	Enable track changes by setting it to true
old_file	Required	The name or pattern (without the extension) to rename the old JSON file to. Daily Example: use `{filepath}_{date}` to save the date when the change was detected. Time Example: use `{filepath}_{datetime}` to save the date and time when the change was detected.
old_folder	Optional	The folder name or pattern to store old JSON files to Same Folder Example: use `{domain}` to save the old JSON file in the "example.com" folder. Archive Folder Example: use `{domain}_{date}` to save the old JSON file in the "example.com_2020-08-25" folder based on when the change was detected.
change_email	Optional	The email address to send change notification and the new and old JSON files to. Use a semicolon ; separated list for multiple recipients such as: `[email protected]; [email protected]; [email protected]`
change_email_subject	Optional	The subject of the email. Defaults to: "Data change detected"

On-Demand Track Changes POST example

The example below is using a comprehensive POST request with:

Azure storage account (azure)
Storage file name patterns (old_folder, old_file)
Change tracking enabled
Email notification enabled

POST https://api.addscreenshots.com/screenshots/?apikey=YOUR_API_KEY

{
  "url": "https://example.com/",
  "azure": "https://screenshots.blob.core.windows.net/screenshotdemo?st=2020-08-10T07%...fEWvNz6fbdeUenY%3D",
  "data": {
    "script": "bGV0..Base64-Encoded-JavaScript...",
    "image": false,
    "pretty": true,
    "track_changes": true,
    "change_email": "[email protected];[email protected]",
    "change_email_subject: "The Data Changed",
    "file": "current",
    "folder": "json-files",
    "old_file": "{date}",
    "old_folder" "history"
  }
}

Your API key can be found on the API Keys page. Need an API Key? Sign up to get started.
For internal applications, replace YOUR_API_KEY with your own unique API Key.
For public facing websites or hotlinks, generate a signed URL.