Skip to content
Colin M. Skinner, PhD

Product engineering · Sep 11, 2026

AI news board

An AI-powered news intelligence platform that aggregates and curates news from 16 sources, tracks emerging technologies and shifts in coverage over time, and visualizes trends through an interactive semantic landscape. Features an interactive voice agent delivering curated news briefings, real-time conversation, and on-demand research.

  • Next.js
  • embeddings
  • UMAP
  • realtime voice

Project Overview

AI News Board is an AI-powered news aggregation and intelligence platform integrated into my personal website. It combines automated news collection, semantic search, longitudinal trend analysis, interactive data visualization, and a real-time conversational voice agent to help users navigate the rapidly evolving artificial intelligence landscape.

The project addresses a fundamental challenge facing anyone working in or following AI: the volume of information is growing faster than an individual can reasonably consume and evaluate.

Research breakthroughs, model releases, new developer tools, infrastructure developments, and industry announcements emerge continuously across dozens of sources. Traditional news aggregators organize this information chronologically, but they provide limited insight into which developments are attracting sustained attention, which are gaining momentum, and which are fading from public discussion.

AI News Board approaches this problem through three complementary capabilities:

  1. Intelligent News Aggregation: Collecting, filtering, ranking, and organizing AI developments from diverse sources.
  2. Longitudinal Trend Intelligence: Identifying specific AI technologies, products, models, and developments, then measuring how attention toward them changes over time.
  3. Conversational News Briefings: Delivering curated spoken news summaries through an interactive AI agent that supports natural interruptions, follow-up questions, live research, and seamless continuation.

Rather than attempting to determine which technological developments are objectively most important, the platform provides users with evidence about where attention is concentrated and how that attention is evolving, allowing them to decide what deserves closer investigation.


Multi-Source News Aggregation

The platform regularly collects AI-related news from 16 configured sources spanning major AI research laboratories, academic publications, technical communities, and industry reporting.

These sources include:

  • Research laboratories and model developers.
  • Academic research repositories and publication feeds.
  • AI-focused technical journalism.
  • Open-source research and development communities.
  • Community-driven technology aggregators.

Automated Ingestion Pipeline

The ingestion system supports multiple source formats, including RSS/Atom feeds, REST APIs, and specialized aggregation endpoints.

Incoming articles pass through a structured processing pipeline:

Source Retrieval → Normalization → Quality Filtering → Per-Source Deduplication → Ranking → Source Balancing → News Board

The system retrieves sources concurrently, applies request timeouts, and handles individual source failures without requiring the entire news board to fail.

Article metadata is normalized into a common representation containing information such as:

  • Article title and URL.
  • Source identity.
  • Publication timestamp.
  • Article summary.
  • Topic classifications.

URLs are canonicalized to reduce duplicate entries caused by tracking parameters or superficial URL differences.

Quality Filtering and Ranking

The platform applies source-aware quality filters to reduce irrelevant or low-value content.

These filters distinguish substantive AI developments from promotional material, unrelated discussions, and other content unlikely to contribute meaningfully to the news experience.

Articles are ranked using a combination of factors, including:

  • Source weighting.
  • Publication recency.
  • Research and technical significance signals.
  • Product and model announcements.
  • Industry developments.
  • Community engagement signals where available.

The ranking system also incorporates source balancing, preventing a single high-volume feed from dominating the board.

The resulting news board presents up to 80 stories, prioritizing recent developments while preserving coverage across different information sources. Reports of the same story from different outlets each remain on the board; they are merged when the spoken briefing is prepared.

Historical Collection

The platform distinguishes between the requirements of a live news feed and those of historical trend analysis.

The live board emphasizes current, relevant information. The trend-analysis pipeline additionally retains qualifying historical articles, including records collected through dedicated backfilling processes.

This separation enables the system to maintain a focused news experience while accumulating the historical evidence necessary to identify changes in attention over time.


Semantic News Search

The news board supports conventional filtering alongside embedding-based semantic search.

Users can search articles by topic, source, and text, while semantic retrieval allows them to discover stories related to a concept even when the article does not contain the exact query terms.

Embedding-Based Retrieval

Article titles and summaries are transformed into vector representations using a configurable embedding model.

These embeddings are stored and reused, reducing unnecessary computation when article text and the configured model remain unchanged.

When a user submits a search query, the system generates a query embedding and compares it against indexed article vectors using cosine similarity.

Results are ranked according to semantic relevance. Semantic matches below a fixed similarity threshold are dropped unless the article also contains the query text.

Hybrid Search and Fallback

The retrieval system combines semantic matching with lexical search.

When embeddings are unavailable, the system falls back to conventional text matching rather than disabling search functionality.

This design maintains basic search availability even when external model services are unavailable or embedding generation is not configured.


AI-Powered Topic Extraction

A central component of the platform is its ability to transform individual news stories into persistent, trackable topics.

Unlike conventional news categorization, which assigns articles to broad categories such as Research, Models, or Industry, the trend-analysis system identifies specific developments that can be monitored across articles and over time.

Recent examples on the live map include:

  • Flux 3 Image
  • Torch Neuron
  • SynthID Detector
  • Cloudflare AI Platform
  • AI Agent Sandboxing
  • Claude Abuse Policy

These topics can represent individual models, developer tools, infrastructure platforms, technical initiatives, or broader developments receiving attention within the AI ecosystem.

Structured Topic Extraction

The system uses a language model to analyze article titles and summaries and extract structured topic information.

For each qualifying article, the extractor can identify up to four topics, along with descriptions and associated entities.

Extracted topics pass through additional validation and filtering.

The system rejects overly generic classifications and applies technology-focused eligibility rules intended to keep the landscape centered on identifiable AI developments.

For example, broad labels such as AI or Research are not sufficient to establish meaningful topic identities.

Topic Identity and Semantic Deduplication

A major challenge in longitudinal trend analysis is maintaining consistent topic identities.

Different articles may refer to the same underlying technology using different names, abbreviations, or terminology.

Without reconciliation, the system could mistakenly treat these references as independent topics, fragmenting their historical coverage.

To address this, the platform maintains persistent topic identities and aliases.

Topic descriptions can be embedded into a shared semantic space, allowing the system to identify highly similar topics and merge synonymous references using a high, fixed similarity threshold.

This process allows related coverage to contribute to a consistent historical record rather than creating redundant topics.

Topic Granularity

The extraction system is designed to balance specificity against interpretability.

The objective is not to represent every technical phrase or narrowly defined research concept as an independent topic.

Instead, it prioritizes recognizable developments that users can meaningfully follow.

This distinction is particularly important for the interactive topic landscape, where excessive topic fragmentation would make the visualization difficult to interpret.


Longitudinal Trend Analysis

The trend-analysis engine is one of the platform's most distinctive technical components.

Rather than simply reporting which topics are popular today, it measures how coverage of individual topics changes across multiple historical windows.

The system maintains persistent records of topic appearances, article associations, source coverage, and historical counts.

Historical Comparison Windows

The platform supports five rolling analysis windows:

WindowPurpose
24 hoursRecent changes in attention
3 daysShort-term developments
7 daysWeekly trends
30 daysSustained monthly attention
90 daysLonger-term attention patterns

Each window is compared against the immediately preceding window of equal duration.

This allows users to examine both rapidly emerging developments and topics exhibiting sustained or declining attention.

Measuring Share of Coverage

The system measures attention using the proportion of qualifying news stories associated with a topic, rather than relying exclusively on raw mention counts.

For a given topic and time window:

Coverage Share=Stories Tagged With TopicTotal Counted Stories in Window\text{Coverage Share} = \frac{\text{Stories Tagged With Topic}} {\text{Total Counted Stories in Window}}

Each qualifying story contributes at most once to a particular topic's count.

Because a story can be associated with multiple topics, coverage shares across different topics do not necessarily sum to 100%.

The system then compares the current coverage share with the preceding window:

Relative Growth=Current Share−Previous SharePrevious Share\text{Relative Growth} = \frac{\text{Current Share} - \text{Previous Share}} {\text{Previous Share}}

This normalization is important because the total volume of collected news can fluctuate.

For example, if the number of articles collected doubles while the proportion discussing a particular technology remains unchanged, that technology should not automatically be classified as gaining attention.

The objective is to identify changes in relative attention, not simply increases in publication volume.

Trend Classification

Topics are assigned visual classifications based on changes in their share of coverage.

ClassificationInterpretation
SurgingCoverage share has at least doubled from a nonzero baseline, with corroboration from at least three sources.
GrowingCoverage share has increased by more than 25%, without meeting the surging criteria. An established topic with no coverage in the previous window and at least three stories now is also shown as Growing, without a percentage.
StableCoverage share has changed by no more than approximately 25% in either direction.
DecliningCoverage share has decreased by more than 25%.
New / No ComparisonThe topic was first seen within the last day, or the previous window was not recorded, so there is no baseline to compare against.

The system also distinguishes newly observed topics from established topics with historical baselines.

These classifications describe changes in news coverage, not independently verified changes in technological importance.

Statistical Measurements

In addition to relative percentage changes, the platform calculates statistical measures associated with changes in topic coverage.

These include a two-proportion z-score, momentum acceleration, and a heuristic evidence-strength score shown as Strong, Some, or Limited evidence.

The current visualization primarily uses percentage-change thresholds for its color classifications rather than directly mapping statistical significance to visual categories.

This separates the system's internal analytical measurements from the simpler signals presented to users.

Historical Coverage Validation

A significant challenge in automated trend analysis is distinguishing genuine changes in attention from incomplete data collection.

A topic might appear to surge simply because historical ingestion was interrupted or a source became temporarily unavailable.

To reduce misleading classifications, the platform tracks successful collection runs and historical coverage.

Analysis windows can remain visually unclassified when the underlying collection history is incomplete, stale, or contains significant gaps.

This ensures that users are not presented with confident trend signals when the system lacks sufficient historical observations.


Interactive AI Topic Landscape

The platform presents trend-analysis results through an interactive spatial visualization.

Each circle represents a tracked AI topic.

The visualization encodes multiple dimensions of information:

  • Circle size: Volume of topic coverage within the selected window.
  • Circle color: Trend classification based on the change in relative coverage.
  • Spatial position: Semantic similarity or thematic organization.
  • Time-window selection: Changes in topic attention across different historical periods.

The primary purpose of the landscape is to help users understand which developments are gaining attention, maintaining attention, or declining over time.

Semantic and Thematic Organization

The visualization combines two related approaches to topic placement.

The first uses semantic embeddings to position related topics near one another, for topics without an assigned theme.

For sufficiently large topic collections, dimensionality reduction using UMAP can contribute to this layout.

The second organizes topics around predefined thematic regions.

The platform defines 12 thematic areas:

  1. AI Agents
  2. Reasoning and Planning
  3. Foundation Models
  4. Multimodal AI
  5. Robotics and Embodied AI
  6. Generative Media
  7. AI Infrastructure
  8. AI Safety and Alignment
  9. AI Research and Methods
  10. AI Policy and Governance
  11. AI Industry and Economics
  12. AI Adoption and Applications

Topics assigned to these themes are positioned around corresponding regions of the landscape.

This provides users with a structured overview of how attention is distributed across the broader AI ecosystem.

Spatial Stability

The system preserves historical topic positions where possible.

This reduces unnecessary movement between updates, helping users develop a consistent mental model of the landscape.

In the semantic layout, existing topic positions are retained or adjusted within bounded movement limits.

In the thematic layout, topics are organized around stable thematic anchors.

Topic Selection and Visual Clarity

The landscape does not attempt to display every extracted topic simultaneously.

Instead, it applies eligibility requirements and selects a bounded mixture of topics, including:

  • Frequently discussed topics.
  • Surging developments.
  • Newly observed topics.
  • Declining topics.
  • Representatives from different thematic regions.

The displayed topic count is capped to maintain readability.

This design balances information density with interpretability, helping users identify meaningful patterns without overwhelming them with hundreds of overlapping elements.

Supporting Evidence

The visualization connects individual topics to the underlying news stories contributing to their coverage.

Users can therefore move from a high-level trend signal to the retained articles associated with that topic.

This provides an important distinction between an unexplained algorithmic classification and a trend indicator that users can investigate themselves.


Conversational AI News Briefing Agent

The platform includes a real-time conversational AI agent that transforms the news board into an interactive spoken briefing.

Rather than requiring users to read through dozens of articles, the agent presents a curated summary of recent developments and allows users to explore individual stories through natural conversation.

Automated Briefing Generation

The system selects up to five significant stories from the most recent available news day.

Before selection, it reconciles duplicate coverage of the same story across different sources.

Stories are ranked using the platform's news-ranking logic.

A language model then generates a concise spoken briefing based on the selected stories.

The generated briefing is cached and reused until the underlying story selection or relevant generation configuration changes.

This reduces redundant model calls and improves responsiveness.

Real-Time Voice Interaction

The briefing is delivered through a real-time conversational AI system supporting bidirectional audio streaming.

Users can listen to the briefing, interrupt the agent, ask questions, and continue the conversation without switching between separate interfaces.

The experience is designed to resemble a conversation with an informed news presenter rather than a conventional text-to-speech reader.

Natural Interruption and Resumption

One of the more challenging interaction problems addressed by the project is preserving continuity when users interrupt a spoken briefing.

For example, a user might interrupt halfway through the third story to ask a question.

After answering, the agent returns to the briefing on its own, and the user can also ask it to continue at any point.

The agent resumes the briefing from the appropriate point rather than restarting from the beginning.

This behavior is implemented through conversational context and carefully designed model instructions, rather than a deterministic audio playback cursor.

The approach avoids introducing a separate mechanism for synchronizing spoken audio progress with application state.

In practice, the prompt-based approach has provided reliable continuation behavior while preserving the responsiveness of the real-time interaction.

Live Research and Follow-Up Questions

The agent is not limited to the stories included in its opening briefing.

It can answer follow-up questions about the news and, when a question goes beyond its story list, retrieve additional AI and tech news through integrated web search.

For example, a user listening to a briefing about a newly released language model might ask:

How does that model compare with the latest competing models?

The agent can conduct additional research, explain the comparison, and subsequently return to the original briefing when requested.

This creates a flexible interaction model combining:

Curated News Briefing → User Interruption → Contextual Discussion → Live Research → Briefing Continuation

The result is a news experience that supports both passive listening and active investigation.

Grounding and Content Integrity

The system includes safeguards intended to keep generated briefings grounded in the selected news stories.

These include structured story boundaries, validation of generated content, and filtering of suspicious instructions embedded in source material.

The objective is to reduce the risk that external news content can redirect the agent's behavior or that generated briefings introduce unsupported claims.

The voice agent also receives explicit instructions governing when to use web search and how to report what it finds: name the publication, never invent headlines, dates, or links, and say so when it cannot confirm something.


Application Architecture and Infrastructure

AI News Board is implemented as a full-stack application using Next.js, TypeScript, server-side processing, persistent data storage, and external AI services.

The architecture separates several workloads with different operational requirements.

Web Application

The Next.js and TypeScript application exposes the live board, topic landscape, semantic search, and voice-briefing interface. Server-side endpoints provide normalized stories, search rankings, topic-specific evidence, and cached briefing data. Browser-side controls handle filtering and visualization without recalculating historical momentum on the client.

Data and Background Processing

News retrieval is concurrency-limited, time-bounded, and fault-tolerant: feeds are fetched in parallel, normalized into validated records, and ranked with per-source limits. A daily scheduled job ingests qualifying articles, reconciles topics, and publishes updated trend snapshots; a separate backfill job loads historical coverage. Production trend history uses PostgreSQL; SQLite supports local and lightweight caches, including briefing and embedding records.

The system distinguishes live presentation, historical recording, and published trend analysis. That separation prevents a temporary feed outage or delayed analysis job from taking down the news board and allows trend windows to be marked incomplete rather than presenting misleading results.

Dedicated Real-Time Voice Service

Real-time audio uses a separate Node.js voice host with a WebSocket upgrade handler rather than running inside a short-lived serverless function. A server-side relay connects the browser to a real-time multimodal model while keeping provider credentials off the client.

The relay validates messages, forwards bidirectional PCM audio and transcription events, relays interruptions detected by the real-time model's voice activity detection, and enforces session duration limits. One-time session identifiers, origin checks, and per-IP and overall usage quotas constrain access to the public demonstration; API sessions are also limited to one live connection per key.

Resilience and Operational Controls

The implementation uses request deadlines, caching, graceful fallbacks, schema validation, guarded API routes, and scheduled maintenance. If live news sources are unreachable, the interface explicitly labels sample headlines. If semantic embeddings are unavailable, search falls back to lexical matching. If a generated briefing fails validation, the application can use a deterministic, story-grounded alternative.


Engineering Challenges and Design Decisions

Preserving Topic Identity Over Time

News articles rarely use consistent terminology. The system therefore treats topic identity as a persistent data-management problem, not merely a one-off classification task. Structured extraction, semantic matching, aliases, and stable topic records prevent related coverage from fragmenting into separate histories.

Avoiding Misleading Trend Signals

Raw article counts are sensitive to changes in source volume. The trend engine instead compares each topic's share of coverage against an equal-length prior window. It also checks source diversity before labeling a topic Surging and withholds confident visual classifications when historical collection coverage is insufficient. These choices prioritize interpretable signals over visually dramatic but unreliable trends.

Making a Dynamic Map Legible

A topic landscape must convey both change and continuity. The map limits the number of visible topics, includes a mix of high-coverage and emerging developments, organizes related topics spatially, and bounds movement of established topics between updates. Users can inspect the articles behind a topic rather than relying on a color alone.

Designing Conversational Continuity

The voice briefing must support both a prepared opening and unscripted follow-up discussion. Instead of building a separate audio-position tracker, the agent uses conversation context and explicit continuation instructions to return to the interrupted briefing. The dedicated WebSocket relay addresses a different challenge: keeping low-latency audio streaming and session control separate from the web application's request-response lifecycle.

Treating External Content as Untrusted

Article text is useful context but is not an instruction source. The briefing pipeline fences story content, filters instruction-like text, and validates generated output against the selected stories. This is an engineering safeguard against prompt injection and unsupported claims, not a guarantee that generated language can never be wrong.


Product Philosophy and User Experience

AI News Board is designed to reduce the effort required to follow a fast-moving technical field without pretending to decide which advances matter most. The live board answers what happened; the topic landscape helps users assess where attention is moving; and the conversational agent supports why it matters through questions, explanation, and further research.

The interface offers multiple depths of engagement: scan a few headlines, search the full board, investigate a topic's historical coverage, or listen to and interrupt a spoken briefing. Trend indicators measure attention in the collected news, not objective scientific progress, technical merit, or future commercial success.


Technical Summary

Stack: Next.js, TypeScript, Node.js, WebSockets, PostgreSQL, SQLite, REST APIs, a scheduled ingestion job, backfill tooling, language-model-assisted extraction, vector embeddings, cosine similarity, and UMAP-supported visualization.

Core engineering systems: Multi-source concurrent ingestion; normalization and quality ranking; hybrid semantic/lexical search; persistent topic reconciliation; historical coverage-share analytics; coverage-quality gating; stable interactive topic layouts; cached, grounded briefing generation; and real-time conversational audio with interruption and resumption.

Scope: 16 configured news sources; a balanced live board of up to 80 stories; five rolling trend windows (24 hours, 3 days, 7 days, 30 days, and 90 days); and an interactive map showing a bounded selection of tracked topics.


Project Significance

This project brings together several areas of software and AI engineering that are often demonstrated separately: ingestion and data quality, semantic information retrieval, entity and topic reconciliation, longitudinal analytics, visualization, and real-time agent interaction.

Its central engineering contribution is the integration of those systems into a coherent product. Users can move from a current headline to a persistent topic history, inspect the evidence behind a trend, and discuss developments through a live conversational interface. The result is both a practical way to explore AI news and a demonstration of full-stack, data-intensive, and applied-AI systems engineering.