Markdown Version | Session Recording

Session Date/Time: 11 Feb 2025 19:00

TOOLS

Summary

This TOOLS Working Group session focused on updates regarding ongoing infrastructure projects, primarily the email services transition, enhancements to Data Tracker robustness for IETF 122 (including the migration to blob storage), and automated certificate management. Discussions also covered IMAP service stability, the liaison management tooling, and an important security fix in the XMLRFC toolchain. A decision was made to reschedule the March meeting.

Key Discussion Points

• Email Services Transition:
  • Migration of core email services (Postfix, Mailman) from ITFA to a new cloud VM is in progress.
  • New addresses have been warming since February 1st, with 50% of outbound traffic currently routed through the new infrastructure via a simple Postfix instance wrapped by HAProxy.
  • The upcoming weekend will involve a temporary full shift of sending back to ITFA to facilitate a lift-and-shift of the full ITFA image to the new VM.
  • The target date of February 24th is set for the final cutover of Postfix, Mailman, and Postorius web services to the new machine, expected to incur a short outage (within a two-hour window, though services may not be down for the entire duration).
  • A minor glitch occurred where some Data Tracker emails were rejected during logging configuration adjustments on the new service but were identified and re-sent, with timestamps reflecting the retransmission date.
• IMAP Service Stability and Security:
  • In early February, the IMAP service experienced unavailability for some users due to a component managing Data Tracker credential logins hanging; the root cause is under investigation, and a workaround is in place.
  • High volumes of password cracking attempts, often with "stupid usernames" (e.g., list addresses), were detected. Measures have been implemented to filter and reject this malicious traffic earlier.
• Data Tracker Robustness for IETF 122:
  • Significant efforts are underway to enhance Data Tracker robustness, especially in preparation for IETF 122.
  • This includes mitigating heavy traffic loads by using Cloudflare's managed challenge techniques and optimizing computationally expensive endpoints (e.g., proceedings views are now recomputed on a slow schedule and served from cache).
  • A transition to blob storage (to replace the problematic shared file system) is in progress, with the code for writing to blob stores currently in development and testing.
  • The initial version will write to both the file system and blob storage. The goal is to deploy views serving draft content directly from blob storage before the IETF 122 draft submission deadline, with complete removal of reliance on the shared file system planned post-IETF 122.
  • A potential minor delay in request processing is anticipated during the initial deployment of writing to blob stores, but significant performance improvements and reduced load on the origin are expected once content is served directly from blob storage.
• Automated Certificate Management:
  • The manual process for managing certificates (Let's Encrypt) for the mail, IMAP, and rsync servers, including associated TLSA record updates, has been automated.
  • The first production run of this automation is currently being observed, with a certificate renewal and TLSA record update having occurred.
  • Monitoring is in place for certificate and TLSA record changes, and further automation for checking alignment between them is planned.
• Liaison Management Tooling:
  • The reimplementation of liaison management tooling within the Data Tracker is currently in the requirements discussion phase, with ongoing conversations and issue tracking on GitHub. This work is scheduled to follow the current Data Tracker robustness enhancements.
• XMLRFC Toolchain Update:
  • XMLRFC 3260 was released to fix a security issue where the allow-local-file-access command option was not respected when processing artwork and source code. Users now need to explicitly use this option to allow local file access when including such content.
• Community Engagement:
  • The chairs invited feedback on how to increase community participation and enhance the usefulness of the TOOLS WG meetings, noting the current staff-to-community member ratio.

Decisions and Action Items

• Decision: The March TOOLS WG meeting will be moved forward by one week to avoid conflicts with IETF 122 travel and to allow for an update on the mail migration and blob storage work before the meeting. (A sense of those present indicated support for this move).
• Action Item: IETF Secretariat staff will send out messaging by the end of this week or early next week regarding the planned email service outage around February 24th.
• Action Item: Further investigation and implementation of automated monitoring for alignment between certificates and TLSA records will continue.
• Action Item: The chairs will explore options for increasing community participation and engagement in TOOLS WG meetings.

Next Steps

• Continue the email service migration, targeting the cutover of core services by February 24th.
• Deploy blob storage writing capabilities to production before the IETF 122 draft submission deadline.
• Investigate and resolve the root cause of the IMAP service hanging issue.
• Continue gathering requirements for the liaison management tooling reimplementation.