Building Knowledge Graphs from Unstructured Web Data

Introduction

Knowledge graphs have become fundamental infrastructure in modern AI systems — powering everything from Google's search results to enterprise data integration. But building one from scratch involves solving several interconnected problems.

This article walks through the complete pipeline of constructing a knowledge graph from unstructured web data.

The Pipeline

text

Web Sources
    ↓
Data Collection (Scraping)
    ↓
Text Preprocessing
    ↓
Entity Extraction (NER)
    ↓
Relationship Identification
    ↓
Graph Construction (Neo4j)
    ↓
Query Interface (Q&A)

Each stage presents unique challenges. Let's break them down.

Stage 1: Data Collection

Web scraping is the foundation. The quality of your knowledge graph is bounded by the quality of your source data.

Key Considerations

python

import requests
from bs4 import BeautifulSoup
import time

def scrape_with_respect(url, delay=1.0):
    """Scrape responsibly with rate limiting and error handling."""
    time.sleep(delay)
    
    headers = {
        'User-Agent': 'KnowledgeGraphBot/1.0 (research)',
    }
    
    try:
        response = requests.get(url, headers=headers, timeout=10)
        response.raise_for_status()
        
        soup = BeautifulSoup(response.text, 'html.parser')
        
        # Extract meaningful text content
        for script in soup(['script', 'style', 'nav', 'footer']):
            script.decompose()
        
        return soup.get_text(separator=' ', strip=True)
    
    except requests.RequestException as e:
        print(f"Failed to scrape {url}: {e}")
        return None

Best Practices

Rate limiting: Respect robots.txt and add delays between requests
Error handling: Network failures are inevitable at scale
Content extraction: Strip navigation, ads, and boilerplate
Deduplication: Same content often appears at multiple URLs

Stage 2: Entity Extraction

Named Entity Recognition (NER) identifies entities — people, organizations, locations, concepts — from raw text.

Approaches

Approach	Accuracy	Speed	Flexibility
Rule-based (regex)	Low	Fast	Limited
Statistical (CRF)	Medium	Medium	Moderate
Neural (BERT/SpaCy)	High	Slow	High
LLM-based	Highest	Slowest	Highest

SpaCy Implementation

python

import spacy

nlp = spacy.load("en_core_web_trf")

def extract_entities(text):
    """Extract named entities with their types."""
    doc = nlp(text)
    
    entities = []
    for ent in doc.ents:
        entities.append({
            'text': ent.text,
            'label': ent.label_,
            'start': ent.start_char,
            'end': ent.end_char,
        })
    
    return entities

Stage 3: Relationship Extraction

This is where entities become connected — transforming isolated facts into a graph.

Dependency Parsing Approach

python

def extract_relationships(doc):
    """Extract subject-verb-object triples from text."""
    triples = []
    
    for token in doc:
        if token.dep_ in ('nsubj', 'nsubjpass'):
            subject = token
            verb = token.head
            
            for child in verb.children:
                if child.dep_ in ('dobj', 'attr', 'prep'):
                    obj = child
                    triples.append({
                        'subject': subject.text,
                        'predicate': verb.text,
                        'object': obj.text,
                    })
    
    return triples

Stage 4: Neo4j Storage

Neo4j's property graph model maps naturally to knowledge graph structures.

Graph Schema

cypher

// Create entity nodes
CREATE (e:Entity {
    name: $name,
    type: $type,
    source: $source_url,
    confidence: $confidence
})

// Create relationships
MATCH (a:Entity {name: $subject})
MATCH (b:Entity {name: $object})
CREATE (a)-[:RELATES_TO {
    predicate: $predicate,
    source: $source_url
}]->(b)

Querying the Graph

cypher

// Find all entities related to a concept
MATCH (a:Entity)-[r]->(b:Entity)
WHERE a.name CONTAINS 'machine learning'
RETURN a, r, b
LIMIT 25

Stage 5: Question Answering

The ultimate test of a knowledge graph is whether it can answer questions.

text

User Query: "What technologies are used in NLP?"
    ↓
Query Parsing → Extract intent + entities
    ↓
Graph Traversal → Find relevant subgraph
    ↓
Answer Generation → Format response

Lessons Learned

Data quality is everything — Garbage in, garbage out applies doubly to knowledge graphs
Start small — Begin with a focused domain before expanding
Entity resolution is hard — "NY", "New York", and "NYC" are the same entity
Relationships need confidence scores — Not all extracted relationships are reliable
Graph maintenance is ongoing — Knowledge changes; your graph must too

Conclusion

Building a knowledge graph is an exercise in information architecture. Each stage — scraping, extraction, relationship mapping, storage, and querying — requires careful design. The result, when done well, is a queryable representation of knowledge that powers intelligent applications.

References

"Knowledge Graphs" by Aidan Hogan et al. (ACM Computing Surveys, 2021)
Neo4j Graph Data Science documentation
SpaCy NER documentation