Skip to content

Conversation

@szporwolik
Copy link
Contributor

@szporwolik szporwolik commented Dec 3, 2025

RSS Feed Caching for DX Calendar and Contest Calendar

Description

This PR implements a cron-based RSS feed caching system for the DX Calendar and Contest Calendar pages, following the existing pattern for other cron jobs.

Why?

Cleaner architecture and most imporant - posibility to quickly display part of the data in other parts of the website in the future. As a side benefit - on my test instance DX Calendar load time was reduced from 7s to 1.5s.

Problem

Previously, both calendar pages fetched RSS data from external sources (ng3k.com and contestcalendar.com) on every page load. This caused:

  • Slow page loads when external sources were unresponsive
  • Unnecessary load on external servers
  • Not perfect user experience during network issues
  • For DX Calendar: additional performance issues due to 4 database queries per DXpedition entry

Solution

Backend RSS Feed Caching

  • New cron job (update_rss_feeds) that runs every 12 hours to fetch and cache RSS feeds - this can be simply improved to add more RSS sources in the future
  • Lockfile mechanism (/tmp/.update_rss_feeds_running) to prevent concurrent executions - following other cron jobs
  • XML validation before caching to ensure only valid RSS data is stored - simplified the display code
  • Debug page integration for manual trigger and status monitoring

DX Calendar Performance Optimization

  • New batch query function get_batch_dxcalendar_statuses() in Logbook_model
  • Reduces database queries from 4×N (4 queries per DXpedition) to just 4 total queries
  • Uses IN clauses to check all callsigns/DXCCs at once

Frontend Improvements

  • Calendar pages now read from cache only (no external fetches)
  • Role-based error messages:
    • Admins: See instruction to trigger update from Debug page
    • Regular users: See message to contact administrator

Migration

Migration 268 adds the new cron job:

  • ID: update_update_rss_feeds
  • Schedule: 0 */12 * * * (every 12 hours)
  • Description: Update RSS feeds for DX Calendar and Contest Calendar

Testing

  1. Run migration to add the cron job
  2. Navigate to Debug page and trigger "RSS Feeds" update
  3. Verify DX Calendar and Contest Calendar load from cache
  4. Check that cache files are created in application/cache/

@szporwolik szporwolik marked this pull request as ready for review December 3, 2025 00:20
@szporwolik szporwolik changed the title [DX Calendar][Contest Calendar] Backend improvments [DX Calendar][Contest Calendar] Backend improvements Dec 3, 2025
@szporwolik szporwolik changed the title [DX Calendar][Contest Calendar] Backend improvements [DX Calendar][Contest Calendar] Backend optimization Dec 3, 2025
@HB9HIL
Copy link
Contributor

HB9HIL commented Dec 3, 2025

Usually here is caching implemented. Imho it's better to only load the data on the first load and cache it instead loading data you may never need. If the caching does not work on your side you may suffering a bug or something.

@int2001
Copy link
Contributor

int2001 commented Dec 3, 2025

same view as fabian here.
Contest- and DXPed-Cal already have a caching logic.
yes - 1st user of day may wait a bit longer.

but cron doesn't solve a possible issue - if there's a network issue - either.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants