Unit 1: Data Mining Foundations & Web Mining Overview

Syllabus:

Data Mining Foundations: Basic concepts in Data Mining, Web mining versus Data mining, Discovering knowledge from Hypertext data.
An Overview of Web Mining: What is Web mining, Web mining taxonomy, Web mining subtasks, issues, challenges.

🎯 PYQ Analysis for Unit 1

High Priority Topics ⭐⭐⭐

1. Web Mining vs Data Mining (2022-Feb, 2022-Dec, 2023, 2024-May, 2024-Dec)
2. Discovering Knowledge from Hypertext Data (2022-Feb, 2023, 2024-May, 2024-Dec)
3. Web Mining Issues & Challenges (2022-Feb, 2022-Jul, 2023, 2024-May, 2024-Dec)
4. Web Mining Subtasks (2022-Feb, 2023)

Medium Priority Topics ⭐⭐

5. Web Mining Taxonomy (2024-May, 2024-Dec)
6. Data Mining Applications (2022-Jul, 2022-Dec)
7. Social Impacts of Data Mining (2022-Jul)

Short Answer Topics ⭐

8. Define Mining (2022-Feb)
9. Hypertext Data (2023, 2024-May)

Section 1: Data Mining Foundations

1.1 Introduction to Data Mining

PYQ: Define mining. (2022-Feb, 1.875 marks)
PYQ: Define data mining. (2022-Jul, 3 marks)
PYQ: What is data mining? (2024-May, 15 marks)

What is Data Mining?

Data Mining is the process of discovering meaningful patterns, correlations, anomalies, and trends from large datasets using statistical, mathematical, and computational techniques. It is also known as Knowledge Discovery in Databases (KDD).

It involves analyzing data from different perspectives and summarizing it into useful information that can help organizations make better decisions, predict future trends, and gain competitive advantages.

Key Characteristics:

• Automatic Discovery - Finds patterns without explicit programming
• Predictive - Predicts future trends based on historical data
• Scalable - Works with large datasets (terabytes/petabytes)
• Actionable - Produces insights that can be acted upon
• Non-trivial - Discovers hidden, previously unknown patterns

Data Mining Process (KDD Process)

┌──────────────────────────────────────────┐
│  KNOWLEDGE DISCOVERY IN DATABASES (KDD)  │
└──────────────────────────────────────────┘
                  │
    ┌─────────────▼───────────────┐
    │        Data Selection       │
    └─────────────┬───────────────┘
    ┌─────────────▼───────────────┐
    │      Data Preprocessing     │
    └─────────────┬───────────────┘
    ┌─────────────▼───────────────┐
    │     Data Transformation     │
    └─────────────┬───────────────┘
    ┌─────────────▼───────────────┐
    │         Data Mining         │
    └─────────────┬───────────────┘
    ┌─────────────▼───────────────┐
    │     Pattern Evaluation      │
    └─────────────┬───────────────┘
          ┌───────▼───────┐
          │   KNOWLEDGE   │
          └───────────────┘

Steps in KDD Process:

1. Data Selection: Identifying relevant data from databases
2. Data Preprocessing: Cleaning, handling missing values, removing noise
3. Data Transformation: Converting data into suitable format (normalization, aggregation)
4. Data Mining: Applying algorithms to extract patterns
5. Pattern Evaluation: Interpreting and validating discovered patterns
6. Knowledge Presentation: Visualizing and presenting results

Data Mining Techniques

Applications of Data Mining

PYQ: Explain how data mining is used in healthcare analysis. (2022-Jul, 8 marks)
PYQ: Application of data mining in healthcare. (2022-Dec, 8 marks)

Data Mining in Healthcare (Detailed)

PYQ: Explain how data mining is used in healthcare analysis. (2022-Jul, 8 marks)

Applications:

1. Disease Diagnosis & Prediction

   • Using patient symptoms and history to predict diseases
   • Early detection of cancer, diabetes, heart disease
   • Example: Predicting COVID-19 severity based on patient data

2. Drug Discovery

   • Identifying potential drug candidates
   • Analyzing drug interactions and side effects
   • Reducing time and cost of clinical trials

3. Treatment Effectiveness

   • Analyzing which treatments work best for specific conditions
   • Personalized medicine based on patient genetics

4. Hospital Resource Management

   • Predicting patient admission rates
   • Optimizing bed allocation and staff scheduling

5. Medical Image Analysis

   • Detecting tumors in X-rays, MRIs, CT scans
   • Using deep learning for image classification

Healthcare Data Mining Applications
│
├──► Diagnosis
│    ├──► Disease prediction
│    ├──► Risk assessment
│    └──► Symptom analysis
│
├──► Treatment
│    ├──► Drug effectiveness
│    ├──► Personalized medicine
│    └──► Treatment planning
│
├──► Operations
│    ├──► Resource allocation
│    ├──► Cost reduction
│    └──► Quality improvement
│
└──► Research
     ├──► Drug discovery
     ├──► Clinical trials
     └──► Epidemiology

Social Impacts of Data Mining

PYQ: What are the social impacts of data mining? (2022-Jul, 7 marks)

Positive Impacts:

Negative Impacts:

Ethical Considerations:

• Transparency: Users should know how their data is used
• Consent: Explicit permission before data collection
• Anonymization: Protecting individual identities
• Fairness: Avoiding discriminatory outcomes
• Accountability: Clear responsibility for data misuse

Spatial Data Mining

PYQ: Define data mining. Discuss spatial data mining. (2022-Dec, 15 marks)

Spatial Data Mining is the process of discovering interesting and useful patterns from large spatial databases (geographic data). It involves analyzing spatial relationships, distributions, and trends in data that has a geographic or spatial component.

Characteristics of Spatial Data:

• Contains location information (coordinates, addresses)
• Has spatial relationships (distance, adjacency, containment)
• Often combined with non-spatial attributes

Spatial Data Mining Techniques:

Applications:

• Urban Planning: Analyzing traffic patterns, zoning decisions
• Environmental Science: Climate change analysis, pollution monitoring
• Epidemiology: Disease spread analysis, healthcare accessibility
• Marketing: Location-based advertising, store placement
• Agriculture: Crop yield prediction, soil analysis

1.2 Basic Concepts in Data Mining

Types of Data

Data Types in Data Mining
│
├──► Structured Data
│    ├──► Relational databases
│    ├──► Spreadsheets
│    └──► CSV files
│
├──► Semi-Structured Data
│    ├──► XML documents
│    ├──► JSON files
│    └──► Email
│
├──► Unstructured Data
│    ├──► Text documents
│    ├──► Images/Videos
│    ├──► Audio files
│    └──► Web pages
│
└──► Time-Series Data
     ├──► Stock prices
     ├──► Sensor data
     └──► Weather data

Data Mining Tasks

Common Data Mining Algorithms

1. Decision Trees (C4.5, CART, ID3)

   • Tree-like model for classification
   • Easy to interpret and visualize

2. K-Means Clustering

   • Partitions data into K clusters
   • Based on distance from centroids

3. Apriori Algorithm

   • Finds frequent itemsets
   • Used for association rule mining

4. Naive Bayes

   • Probabilistic classifier
   • Based on Bayes' theorem

5. Support Vector Machines (SVM)

   • Finds optimal hyperplane for classification
   • Effective for high-dimensional data

6. Neural Networks

   • Mimics human brain structure
   • Good for complex pattern recognition

1.3 Web Mining vs Data Mining

PYQ: Write down the difference between data mining and web mining. (2022-Feb, 7 marks)
PYQ: Web mining versus data mining. (2022-Dec, 7 marks; 2023, 7.5 marks)
PYQ: Discuss the differences and similarities between web mining and data mining. Also, list various applications of web mining. (2024-Dec, 8 marks)

Web Mining and Data Mining are closely related fields, but they focus on different types of data and have distinct challenges.

Comparison Table

Key Differences

┌─────────────────────────────────────────────────────────────────┐
│                    DATA MINING vs WEB MINING                    │
├─────────────────────────────┬───────────────────────────────────┤
│         DATA MINING         │           WEB MINING              │
├─────────────────────────────┼───────────────────────────────────┤
│ • Structured data           │ • Unstructured/semi-structured    │
│ • Databases/Warehouses      │ • Web pages/Logs/Links            │
│ • Controlled environment    │ • Open, distributed environment   │
│ • Known schema              │ • No fixed schema                 │
│ • Quality data              │ • Noisy, redundant data           │
│ • Slower updates            │ • Rapidly changing                │
│ • Domain-specific           │ • Cross-domain                    │
│ • SQL-based queries         │ • Crawlers and parsers            │
└─────────────────────────────┴───────────────────────────────────┘

Similarities

1. Pattern Discovery: Both aim to find hidden patterns
2. Algorithms: Many algorithms are shared (clustering, classification)
3. Knowledge Extraction: Both convert data into actionable knowledge
4. Preprocessing Required: Both need data cleaning and transformation
5. Iterative Process: Both follow iterative refinement
6. Business Value: Both provide competitive advantages

Why Web Mining is More Challenging

1.4 Discovering Knowledge from Hypertext Data

PYQ: How to discover knowledge from hypertext data? Discuss in detail with a suitable example. (2022-Feb, 8 marks)
PYQ: Discovering knowledge from hypertext data. (2023, 7.5 marks)
PYQ: Explain the process to discover knowledge from hypertext data. (2024-May, 15 marks)
PYQ: How is knowledge discovered from hypertext data, and what are the key challenges involved in the process? (2024-Dec, 7 marks)

What is Hypertext Data?

PYQ: Write short notes on Hypertext data. (2023, 2.5 marks; 2024-May, 2.5 marks)

Hypertext is text displayed on a computer or electronic device with references (hyperlinks) to other text that the reader can immediately access. Hypertext data includes:

• HTML Documents: Web pages with text, images, and links
• Hyperlinks: Connections between documents
• Anchor Text: Clickable text in hyperlinks
• Metadata: Title, keywords, descriptions
• Structure: DOM (Document Object Model) hierarchy

Hypertext Document Structure
│
├──► Content Elements
│    ├──► Text content
│    ├──► Images
│    ├──► Tables
│    └──► Forms
│
├──► Structural Elements
│    ├──► Headers (H1-H6)
│    ├──► Paragraphs
│    ├──► Lists
│    └──► Divisions
│
├──► Link Elements
│    ├──► Internal links
│    ├──► External links
│    ├──► Anchor text
│    └──► Navigation menus
│
└──► Metadata
     ├──► Title
     ├──► Description
     ├──► Keywords
     └──► Author

Knowledge Discovery Process from Hypertext

The process of discovering knowledge from hypertext data involves several steps:

┌────────────────────────────────────────────────────┐
│       KNOWLEDGE DISCOVERY FROM HYPERTEXT DATA      │
└────────────────────────────────────────────────────┘
                         │
         ┌───────────────▼───────────────┐
         │  Data Collection (Crawling)   │
         └───────────────────────────────┘
                         │
         ┌───────────────▼───────────────┐
         │ Data Preprocessing & Cleaning │
         └───────────────────────────────┘
                         │
         ┌───────────────▼───────────────┐
         │      Feature Extraction       │
         └───────────────────────────────┘
                         │
         ┌───────────────▼───────────────┐
         │   Data Analysis & Mining      │
         └───────────────────────────────┘
                         │
         ┌───────────────▼───────────────┐
         │      Pattern Discovery        │
         └───────────────────────────────┘
                         │
         ┌───────────────▼───────────────┐
         │   Knowledge Representation    │
         └───────────────────────────────┘

Step-by-Step Process:

Step 1: Data Collection (Web Crawling)

• Use web crawlers/spiders to fetch web pages
• Follow hyperlinks to discover new pages
• Store HTML content for processing

Web Crawler Architecture
│
├──► Seed URLs
│    └──► Initial list of URLs to crawl
│
├──► URL Frontier
│    └──► Queue of URLs to be visited
│
├──► Fetcher
│    └──► Downloads web pages
│
├──► Parser
│    └──► Extracts links and content
│
└──► Repository
     └──► Stores crawled pages

Step 2: Data Preprocessing

• HTML Parsing: Extract text from HTML tags
• Noise Removal: Remove ads, navigation, boilerplate
• Tokenization: Break text into words/tokens
• Stop Word Removal: Remove common words (the, is, at)
• Stemming/Lemmatization: Reduce words to root form

Step 3: Feature Extraction

Step 4: Pattern Discovery

Apply mining techniques to discover:

1. Content Patterns

   • Topic extraction
   • Keyword clustering
   • Document classification

2. Link Patterns

   • Hub and authority pages
   • Community detection
   • Link prediction

3. Usage Patterns

   • Navigation paths
   • User sessions
   • Click patterns

Step 5: Knowledge Representation

• Ontologies: Formal knowledge representation
• Knowledge Graphs: Entity-relationship networks
• Taxonomies: Hierarchical classification
• Rules: If-then patterns

Example: Knowledge Discovery from News Websites

Scenario: Discovering trending topics from news websites

Step 1: Crawl News Sites
        ├──► CNN, BBC, Reuters, etc.
        └──► Collect articles, headlines, links

Step 2: Preprocess Data
        ├──► Extract article text
        ├──► Remove ads and navigation
        └──► Tokenize and clean text

Step 3: Extract Features
        ├──► Keywords (TF-IDF)
        ├──► Named entities (people, places)
        └──► Categories (politics, sports)

Step 4: Discover Patterns
        ├──► Topic modeling (LDA)
        ├──► Trend detection
        └──► Sentiment analysis

Step 5: Generate Knowledge
        ├──► "Technology" trending this week
        ├──► "Climate change" frequently linked to "politics"
        └──► Positive sentiment around "sports events"

Challenges in Knowledge Discovery from Hypertext

PYQ: What are the key challenges involved in the process? (2024-Dec, 7 marks)

Algorithms for Hypertext Knowledge Discovery

1. PageRank

   • Measures page importance based on links
   • Used by Google for search ranking

2. HITS (Hyperlink-Induced Topic Search)

   • Identifies hubs (link to many) and authorities (linked by many)
   • Useful for topic-specific searches

3. TF-IDF (Term Frequency-Inverse Document Frequency)

   • Measures word importance in documents
   • Used for keyword extraction

4. Latent Dirichlet Allocation (LDA)

   • Topic modeling algorithm
   • Discovers hidden topics in documents

Section 2: Overview of Web Mining

2.1 What is Web Mining?

PYQ: What do you mean by web mining? What are its types? (2022-Feb, 8 marks)
PYQ: Define web mining. Explain its various issues and challenges. (2022-Jul, 15 marks)
PYQ: Write short notes on Web mining. (2022-Dec, 3 marks; 2024-May, 2.5 marks)

Definition of Web Mining

Web Mining is the application of data mining techniques to extract knowledge from web data, including web content, web structure, and web usage data.

"Web Mining is the use of data mining techniques to automatically discover and extract information from web documents and services." — Etzioni (1996)

Why Web Mining?

Components of Web Mining

Web Mining Components
│
├──► Data Sources
│    ├──► Web pages (HTML, XML)
│    ├──► Server logs
│    ├──► User profiles
│    ├──► Hyperlinks
│    └──► Metadata
│
├──► Techniques
│    ├──► Information retrieval
│    ├──► Natural language processing
│    ├──► Machine learning
│    ├──► Database querying
│    └──► Statistical analysis
│
└──► Applications
     ├──► Search engines
     ├──► Recommendation systems
     ├──► Personalization
     ├──► Web analytics
     └──► Sentiment analysis

2.2 Web Mining Taxonomy

PYQ: Explain web mining taxonomy, its issues, and challenges. (2024-May, 15 marks)

What is Web Mining Taxonomy?

Web Mining Taxonomy classifies web mining into different categories based on the type of data being mined and the techniques used. It helps in understanding the various aspects of web mining and their specific applications.

Classification of Web Mining

Web Mining is categorized into three main types based on the type of data being mined:

┌───────────────────────────────────────────────────┐
│                 WEB MINING TAXONOMY               │
└─────────────────────────┬─────────────────────────┘
      ┌───────────────────┼───────────────────┐
┌─────▼─────┐       ┌─────▼─────┐       ┌─────▼─────┐
│   WEB     │       │   WEB     │       │   WEB     │
│ CONTENT   │       │ STRUCTURE │       │  USAGE    │
│  MINING   │       │  MINING   │       │  MINING   │
└─────┬─────┘       └─────┬─────┘       └─────┬─────┘
┌─────┴─────┐       ┌─────┴──────┐      ┌─────┴─────┐
│• Text     │       │• Hyperlinks│      │• Server   │
│• Images   │       │• Document  │      │  Logs     │
│• Audio    │       │  Structure │      │• Cookies  │
│• Video    │       │• Web Graph │      │• Sessions │
└───────────┘       └────────────┘      └───────────┘

1. Web Content Mining (WCM)

Definition: Mining the content of web pages (text, images, audio, video).

Data Sources:

• HTML/XML documents
• Text content
• Multimedia (images, videos, audio)
• Structured data (tables, lists)

Techniques:

Applications:

• Search engines (Google, Bing)
• Question answering systems
• Document summarization
• Content categorization

2. Web Structure Mining (WSM)

Definition: Mining the hyperlink structure of the web to discover useful patterns.

Data Sources:

• Hyperlinks (inlinks, outlinks)
• Anchor text
• Document structure (DOM)
• Site structure

Techniques:

Applications:

• Search engine ranking
• Web page classification
• Finding authoritative sources
• Detecting web spam

3. Web Usage Mining (WUM)

Definition: Mining user access patterns from web server logs and user data.

Data Sources:

• Web server logs
• Proxy server logs
• Browser cookies
• User profiles
• Click-through data

Techniques:

Applications:

• Personalization
• Recommendation systems
• Website optimization
• User behavior prediction

Comparison of Web Mining Types

2.3 Web Mining Subtasks

PYQ: What are web mining subtasks? Discuss in detail with a suitable example. (2022-Feb, 7 marks)
PYQ: Explain web mining subtasks, issues, and challenges. (2023, 15 marks)

Overview of Web Mining Subtasks

Web Mining consists of several interconnected subtasks that work together to extract knowledge from web data:

Web Mining Subtasks
│
├──► Resource Discovery
│    └──► Finding relevant web resources
│
├──► Information Selection & Preprocessing
│    └──► Selecting and preparing data
│
├──► Generalization
│    └──► Discovering patterns
│
└──► Analysis
     └──► Interpreting results

Subtask 1: Resource Discovery

Purpose: Locating and retrieving relevant web documents and resources.

Activities:

• Web Crawling: Automated traversal of the web to collect pages
• Focused Crawling: Targeting specific topics or domains
• Deep Web Access: Retrieving content behind forms and databases
• API Integration: Collecting data from web services

Tools:

• Web crawlers (Scrapy, Apache Nutch)
• Search engine APIs (Google, Bing)
• Web scraping tools (BeautifulSoup, Selenium)

Subtask 2: Information Selection and Preprocessing

Purpose: Extracting and cleaning relevant information from collected resources.

Activities:

Example Process:

    Raw HTML Page
         │
         ▼
┌──────────────────┐
│ Remove HTML tags │
│ Remove scripts   │
│ Remove CSS       │
└────────┬─────────┘
         │
         ▼
┌──────────────────┐
│ Extract text     │
│ Tokenize         │
│ Remove stopwords │
└────────┬─────────┘
         │
         ▼
┌──────────────────┐
│ Stemming         │
│ Feature vectors  │
│ TF-IDF scores    │
└──────────────────┘

Subtask 3: Generalization (Pattern Discovery)

Purpose: Applying data mining algorithms to discover patterns and knowledge.

Techniques Used:

Example: Clustering news articles by topic

Input: 1000 news articles
    │
    ▼
K-Means Clustering (k=10)
    │
    ▼
Output: 10 topic clusters
├── Cluster 1: Politics (150 articles)
├── Cluster 2: Sports (200 articles)
├── Cluster 3: Technology (120 articles)
├── Cluster 4: Entertainment (180 articles)
└── ... and so on

Subtask 4: Analysis and Validation

Purpose: Interpreting discovered patterns and validating their usefulness.

Activities:

• Pattern Evaluation: Assessing pattern quality and significance
• Visualization: Presenting results graphically
• Validation: Verifying patterns against domain knowledge
• Interpretation: Understanding what patterns mean
• Application: Using knowledge for decision-making

Metrics for Evaluation:

• Support: How frequently a pattern occurs
• Confidence: Reliability of the pattern
• Lift: Strength of association
• Precision/Recall: Accuracy of classification

2.4 Web Mining Issues and Challenges

PYQ: Define web mining. Explain its various issues and challenges. (2022-Jul, 15 marks)
PYQ: What are the key issues in web mining? (2022-Feb, 8 marks)
PYQ: Discuss the key issues and challenges faced in web mining. (2024-Dec, 8 marks)

Major Issues in Web Mining

While web mining offers significant opportunities, it also presents several challenges:

Web Mining Issues
│
├──► Data-Related Issues
│    ├──► Volume (scale)
│    ├──► Variety (heterogeneity)
│    ├──► Velocity (dynamism)
│    └──► Veracity (quality)
│
├──► Technical Issues
│    ├──► Crawling efficiency
│    ├──► Storage requirements
│    ├──► Processing complexity
│    └──► Algorithm scalability
│
├──► Semantic Issues
│    ├──► Ambiguity
│    ├──► Context understanding
│    ├──► Multilingual content
│    └──► Synonymy/Polysemy
│
└──► Ethical Issues
     ├──► Privacy concerns
     ├──► Copyright violations
     └──► Data misuse

Detailed Challenges

Technical Challenges in Detail

1. Web Crawling Challenges

2. Data Quality Challenges

Data Quality Problems
│
├──► Incomplete Data
│    └──► Missing pages, broken links
│
├──► Inconsistent Data
│    └──► Same entity, different representations
│
├──► Inaccurate Data
│    └──► Outdated, incorrect information
│
├──► Noisy Data
│    └──► Ads, boilerplate, irrelevant content
│
└──► Biased Data
     └──► Over-representation of popular sites

3. Semantic Challenges

Solutions to Web Mining Challenges

2.5 Applications of Web Mining

PYQ: What are some common applications of web mining? How do they benefit from web mining techniques to improve decision-making processes? (2024-Dec, 7 marks)
PYQ: List various applications of web mining. (2024-Dec, 8 marks)

Major Applications

Application Examples

1. Search Engine Optimization (SEO)

• Analyzing web structure for ranking
• Keyword analysis and optimization
• Competitor analysis

2. Recommendation Systems

• Product recommendations (Amazon)
• Content recommendations (Netflix, YouTube)
• Friend suggestions (Facebook, LinkedIn)

3. Web Analytics

• User behavior analysis
• Conversion optimization
• A/B testing insights

4. Competitive Intelligence

• Market trend analysis
• Competitor monitoring
• Price comparison

5. Customer Relationship Management (CRM)

• Customer profiling
• Churn prediction
• Personalized marketing

Summary Table: Unit 1 Key Concepts

Quick Revision: Important Definitions

Expected Questions for Exam

15 Marks Questions

1. Web Mining vs Data Mining (with comparison table)
2. Knowledge Discovery from Hypertext Data (with process diagram)
3. Web Mining Taxonomy (all three types in detail)
4. Web Mining Issues and Challenges (comprehensive list)

7-8 Marks Questions

1. Applications of Data Mining in Healthcare
2. Web Mining Subtasks
3. Social Impacts of Data Mining
4. Spatial Data Mining

2.5-3 Marks Questions

1. Define Mining / Data Mining / Web Mining
2. What is Hypertext Data?
3. Types of Web Mining
4. Any two challenges in Web Mining

These notes were compiled by Deepak Modi
Last updated: December 2025