Does web data integration require scraping pages?

No. APIs, bulk downloads, feeds, and structured datasets may provide suitable inputs. The access method depends on the source and intended use.

Can an AI model resolve every conflict?

It can assist with proposals, but missing evidence or incompatible definitions may leave a conflict unresolved. A fluent explanation is not verification.

Must all data be copied into one database?

No. Some systems combine copied datasets, while others query sources through a common interface. Either design needs clear identity, meaning, and freshness rules.

AI Glossary / Data engineering and architecture / Intermediate

Web data integration

Definition

Web data integration is the process of bringing information from different web-accessible sources into a coherent form for use together. It includes acquiring the data, understanding its meaning, matching related records, resolving differences, and maintaining the combined result. The sources may include APIs, downloadable datasets, structured web content, and pages whose information needs extraction.

What it means

Collecting several files is only the beginning. One website may report a store's opening hours, another its geographic location, and another customer-service information. Integration establishes which records refer to the same store and how their fields should fit together.

For an AI application, this can produce a searchable knowledge collection, an analytical dataset, or features for a model. The central challenge is not simply obtaining more material. It is combining material without silently changing meaning, duplicating entities, or hiding conflicting and outdated information.

How it works

First identify suitable sources and their access methods, update schedules, permissions, and limitations. Prefer documented structured interfaces when they provide the needed information. Preserve source identifiers and retrieval details as data enters the pipeline.

Next, parse each source into a consistent structure. Map fields, normalize formats, and resolve entities using evidence such as stable identifiers. Compare units, dates, categories, and definitions before combining values. A similar name is not a reliable universal identifier.

Finally, validate the integrated data, record unresolved conflicts, and publish it to the intended application. Monitor source changes and rerun or revise transformations when necessary. Integration is an ongoing process because web pages, schemas, and underlying facts can change independently.

Three practical examples

1. Local-service search: A community directory combines official location data with service descriptions, preserving the source and update date for each organization.

2. Product analysis: A business aligns suppliers' product specifications using manufacturer identifiers and explicit unit conversions, while retaining uncertain matches for review.

3. Research assistance: An AI search tool connects publication metadata from several repositories and links duplicate records without treating repeated descriptions as independent evidence for a claim.

Think of it this way

Think of assembling a travel itinerary from train schedules, hotel bookings, and event details. The pieces become useful together only when dates, locations, and time zones agree.

The analogy highlights coordination across sources. Its limit is that a web integration pipeline must perform those checks repeatedly and at scale, including when a source changes without notice. A readable combined itinerary is not enough if the underlying arrival time came from an obsolete schedule.

Worked example

Suppose source A lists 100 facilities using official facility identifiers. Source B lists 80 records, of which 60 carry identifiers matching A and 20 have new identifiers. Assuming each source has no internal duplicates and the identifiers have the same scope, the combined collection contains 120 distinct facilities: 100 plus 80 minus 60.

Simply appending the files would produce 180 rows and overcount the facilities. Now suppose ten matched facilities have different opening hours. Deduplication alone cannot decide which hours are correct. The pipeline records both claims with their source and effective date, then applies an agreed resolution policy or flags the conflict.

The count calculation depends on the stated identity assumptions. If identifiers are reused or missing, matching becomes a separate evidence problem rather than simple subtraction.

Illustrative case study

Fictional case: Web Orchard builds a directory assistant from municipal facility pages and an open-data download. Its first pipeline merges records by name and keeps whichever opening-hours value arrives last.

Two facilities called Riverside Centre are incorrectly merged, while an older page overwrites a newer schedule. The team switches to official identifiers where available, adds location checks for remaining matches, and distinguishes retrieval time from the date the information became effective. The assistant shows unresolved schedule conflicts instead of inventing one definitive answer.

The scenario illustrates integration design choices. It is not a claim about a real municipality, measured accuracy improvement, or a particular commercial tool.

Common misconceptions and mistakes

A frequent mistake is equating web scraping with complete integration. Extraction obtains information; integration also reconciles structure, identity, meaning, and change.

Another mistake is treating the newest download as the newest fact. A recently retrieved page can describe an old reporting period. Repetition across websites also does not prove independent corroboration when those sites copied one original source.

Finally, a common output format does not establish common semantics. Two columns named revenue may cover different currencies, periods, or accounting definitions. Keep those distinctions explicit before analyzing the combined values.

Business use

Businesses use web data integration for discovery, monitoring, comparison, and enrichment of internal information. Its value depends on whether the combined data supports a defined decision more reliably than the individual sources.

Useful measures include successful source updates, unresolved entity matches, stale records, conflicting values, and the proportion of outputs with traceable origins. Budget for maintenance as well as initial construction. A source redesign or changed identifier policy can disrupt a pipeline even when the receiving AI application has not changed at all.

Check your understanding

A page retrieved today reports figures for an older year, while yesterday's API response reports this year. Which retrieval is necessarily more current in meaning?

Answer: Neither retrieval date alone decides that. Compare the reporting period and effective date of the facts, not just when the pipeline downloaded them.

Related terms

Web scraping
Linked data
Data Quality
Data integration
API
entity resolution
schema mapping
data provenance

Learn next

Combine two small public datasets with a shared identifier. Record source URLs, reporting periods, and retrieval dates. Count unmatched identifiers and inspect conflicting fields before producing a summary. Then describe how you would detect a source change during the next update.

Sources

Sources: W3C, Data on the Web Best Practices: https://www.w3.org/TR/dwbp/. Read for heterogeneous web data, metadata, identifiers, interoperability, and the importance of describing provenance and usage conditions. This entry applies those principles to an original introductory integration workflow; the worked example and fictional case are not W3C case studies.

Previous: Weak supervision Next: Web intelligence