How to Build a Competitor Price Monitoring System Using Python

Imagine running an online store with 2,000 products. Every morning, your competitors adjust prices, introduce discounts, and update their stock availability.

Checking every product manually is nearly impossible.

A competitor price monitoring system helps businesses collect permitted product information, compare prices, and identify important market changes.

In this guide, we'll explore how to build a basic monitoring workflow using Python, from collecting data to detecting price changes.

1. Understand the Monitoring Workflow

Before writing code, define what information you need.

For most e-commerce monitoring projects, the important fields include:

• Product name
• Product identifier (SKU, EAN, UPC, or MPN)
• Retailer name
• Current price
• Currency
• Stock availability
• Collection timestamp

A typical workflow looks like this:

Product URLs → Data Collection → Validation → Database → Price Comparison → Alerts

Each stage helps improve the reliability of the final output.

2. Prepare Your Python Environment

Install Python 3 and the libraries needed for basic HTML parsing.

For this example, we'll use Requests and Beautiful Soup.

pip install requests beautifulsoup4

Requests retrieves HTML documents, while Beautiful Soup helps extract information from their structure.

Only collect data from sources where you have permission. Review the site's terms, access rules, and technical restrictions before implementing automated requests.

3. Extract Product Information

The following example demonstrates HTML parsing using a sample product page stored locally.

This approach lets you understand the extraction process without sending requests to a real retailer.

from bs4 import BeautifulSoup

html = """
<html>
  <body>
    <h1 class="product-title">Wireless Headphones</h1>
    <span class="product-price">49.99</span>
    <span class="stock-status">In Stock</span>
  </body>
</html>
"""

soup = BeautifulSoup(html, "html.parser")

name = soup.select_one(".product-title").get_text(strip=True)
price = float(soup.select_one(".product-price").get_text(strip=True))
stock = soup.select_one(".stock-status").get_text(strip=True)

product = {
    "name": name,
    "price": price,
    "currency": "USD",
    "stock_status": stock
}

print(product)

Expected output:

{
    'name': 'Wireless Headphones',
    'price': 49.99,
    'currency': 'USD',
    'stock_status': 'In Stock'
}

In a production project, extraction logic needs to account for missing fields, changing HTML structures, and validation errors.

For external data sources, businesses can also use official APIs or licensed feeds where available.

4. Store Product Prices for Comparison

Collecting a price once is useful, but historical records provide more meaningful information.

For example, a product priced at $49 today may have cost $55 yesterday.

Python's built-in SQLite library provides a simple way to store records.

import sqlite3

connection = sqlite3.connect("price_history.db")
cursor = connection.cursor()

cursor.execute("""
CREATE TABLE IF NOT EXISTS price_history (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    product_name TEXT,
    retailer TEXT,
    price REAL,
    collected_at TEXT
)
""")

cursor.execute("""
INSERT INTO price_history
(product_name, retailer, price, collected_at)
VALUES (?, ?, ?, datetime('now'))
""", ("Wireless Headphones", "Example Store", 49.99))

connection.commit()
connection.close()

For larger systems, PostgreSQL or another suitable database may offer more flexibility.

5. Detect Price Changes

Once historical prices are available, the next step is identifying changes.

previous_price = 55.00
current_price = 49.99

change = (
    (current_price - previous_price) / previous_price
) * 100

print(f"Price change: {change:.2f}%")

Output:

Price change: -9.11%

A negative result indicates a price decrease.

You can extend this logic to identify price drops beyond a predefined threshold, such as 5% or 10%.

The threshold should be based on business requirements rather than applied equally to every product.

6. Match Products Correctly

Product matching is one of the biggest challenges in competitor price monitoring.

Two retailers may use slightly different names for the same product.

For example:

Retailer A: Wireless Headphones Model X, Black

Retailer B: Model X Bluetooth Headphones – Black Edition

These may be the same product, but name similarity alone is not enough to confirm a match.

A more reliable approach combines available attributes, including:

• Brand
• Manufacturer part number
• GTIN, EAN, or UPC
• Product model
• Pack quantity
• Product specifications

Incorrect matching can lead to misleading price comparisons.

7. Add Stock Availability Monitoring

Pricing is only one part of competitive analysis.

Stock availability provides useful context.

For example, a competing retailer may offer a lower price but have no inventory available.

A monitoring system can store stock status alongside pricing records.

Example:

Businesses can use this information to understand which competing offers are actually available to customers.

8. Generate Reports and Alerts

Once product information is collected and validated, the system can generate structured reports.

Useful reports include:

• Daily competitor price changes
• Products priced above competitors
• Products currently out of stock
• Historical pricing trends
• Significant promotional changes

For larger product catalogs, scheduled workflows can update the data at an appropriate frequency.

The monitoring schedule should respect source permissions, access limits, and operational requirements.

9. Challenges When Scaling a Monitoring System

A basic script may work well for a small number of products.

However, monitoring thousands of products introduces additional challenges.

Data consistency: Product fields and formats may differ across sources.

Source reliability: APIs and permitted data feeds may occasionally be unavailable.

Product matching: Incorrect model or pack-size comparisons can affect reporting accuracy.

Performance: Large datasets require efficient storage and processing.

Data governance: Licensing, source permissions, and privacy requirements must be considered.

Monitoring accuracy: Alerts should distinguish genuine market changes from extraction errors.

A reliable production solution needs proper validation, error handling, logging, and monitoring.

10. From Python Scripts to Business Intelligence

Python provides a useful foundation for understanding competitor monitoring.

But businesses often need more than a single script.

They may require structured datasets, scheduled updates, historical comparisons, and integrations with reporting systems.

Teams can build these capabilities internally or evaluate specialist data providers based on their technical needs and authorized data access.

For businesses exploring product intelligence and competitor monitoring solutions, NenoData provides information about data extraction, e-commerce intelligence, and market monitoring services.

Conclusion

Competitor price monitoring combines several important disciplines: data collection, product matching, validation, historical analysis, and reporting.

Python makes it possible to build a simple monitoring prototype and understand the underlying workflow.

For larger projects, the priority should be reliable data quality, correct product comparisons, permitted source access, and useful business insights.

A well-designed monitoring system helps teams understand market changes without relying entirely on manual research.