Markdown Version | Session Recording

Session Date/Time: 15 Apr 2025 18:00

TOOLS

Summary

The TOOLS working group meeting provided a comprehensive update on several critical initiatives stemming from the recent Tools Retreat. Discussions focused on tools modernization, infrastructure changes (including a potential migration to Azure), and strategies for managing the increasing load from automated systems. Key outcomes included decisions on encoding standards for documents, plans for enhancing mail services, and progress on various development projects.

Key Discussion Points

• IETF 122 Takeaways:
  • IETF 122 operated smoothly from a tools perspective with no major outages.
  • A minor user interface issue was noted where clicking on materials in the Data Tracker agenda opened them in the same popup window, preventing easy navigation back. Internal discussions are underway to address this.
• Tools Retreat Overview: The recent Tools Retreat involved the entire tools team, IETF leadership, and communications directors, covering a wide range of topics.
  • Roadmap Evolution: The current roadmap presentation format is reaching its limits. The team will explore new visualization mechanisms to better represent projects, including those spanning multiple IETF intervals, and provide more detailed insights.
  • Tools Modernization Priority: This project is deemed the most pressing, with a focus on completion before the RFC10K publication in March of next year. Progress is being made on the website, the RPC's workflow management tool (project "Purple"), and an updated ARA system to address current pain points (e.g., authentication, reopening ARA). This suite of applications will be a significant focus leading up to IETF 123.
  • Feature Request Acceleration: Resources will be allocated in parallel to address accumulated feature requests from the ISG, IESG liaison management tooling, and the re-initiation of a cross-property search engine to replace existing search functionalities and provide ISG dashboards.
  • Data Tracker & Mailman 3 Integration: A project is planned to integrate Data Tracker with Mailman 3, allowing basic list subscription management and moderation activities directly within the Data Tracker, using Mailman 3 credentials.
  • Managing Automated Systems/Crawlers: The IETF infrastructure is experiencing a high volume of automated traffic, making it difficult to differentiate between benign crawlers, overly aggressive bots, or malicious attacks.
    • The team will adjust its posture by leveraging a newly hired organization ("new machine futures") to help manage this interrupt load and restore development efficiency.
    • Discussion highlighted the widespread nature of this problem across other systems, with participants speculating that many are "sloppily written AI crawlers."
    • Increased and smarter use of Cloudflare's managed challenge mechanisms (JavaScript-based) will be employed, acknowledging potential user crankiness.
    • Interest was expressed in gathering and sharing data on bot behavior for a potential MAP session, distinguishing between well-behaved bots, those trying to hide, and actual attacks.
  • Authentication Service Separation: The plan to split authentication out of the Data Tracker is moving forward, with "Authentic" (an open-source tool) identified as a suitable candidate. This will enable robust features like multi-factor authentication, passkeys, and integration with other identity providers. The separation is targeted between IETF 123 and 124, requiring customization for Data Tracker's current handling of multiple email addresses per user.
  • Block List Review and Release: The current block lists for IP addresses (firewall, Cloudflare) and email addresses (Postfwd, Postfix) are add-only and growing. New processes will be implemented to periodically release entries, monitor their re-access, and automate removal to prevent lists from becoming excessively large or stale.
  • Artifact Storage Model Adjustment: The authoritative store for artifacts (e.g., RFCs, Internet Drafts) will revert from blob storage to a database directly under IETF control, with blob stores serving as a rebuildable, contained copy.
  • Subservice Decoupling: Several subservices, such as RabbitMQ and Memcache, which were tightly bundled with the Data Tracker, will be moved into separate management environments with their own lifecycles to enable independent, zero-downtime rolling upgrades. This move is driven by an upcoming major RabbitMQ release.
  • Production Migration (Digital Ocean to Azure): A significant project is underway to incrementally migrate IETF production services from Digital Ocean to Azure, aiming for faster compute and better control over database upgrades.
    • A participant raised concerns about the cost, quality, and limited control (e.g., random upgrades, lack of query logs) of Azure's managed database services, suggesting self-hosting on a VM as an alternative. The team acknowledged the need to double-check database access levels.
    • Mail services will continue to be hosted separately at Panix. All front-end services will maintain IPv4 and IPv6 availability.
  • Backup Strategy Enhancement: Current backup strategies will be enhanced to ensure recent backups are available in at least two different clouds, and long-term backups are stored in radically different administrative containers (e.g., different countries) for improved risk mitigation.
  • Zulip Upgrade: The Zulip chat platform will be upgraded from version 8 to 9, introducing user experience changes such as "streams" being renamed to "channels" and UI adjustments for threading. The Meetecho integration has been tested with the new version.
  • BIB Service Modernization: The bib.ietf.org service has stale datasets due to reliance on older Relaton data crawlers. The plan is to adopt current crawlers, map YAML output to XML, store the XML in blob stores for quick access, and integrate with the new cross-property search engine to replace the existing search front end. This aims for easier maintenance and faster data serving.
  • Code Formatting: The community request to adopt black for Python code formatting in the Data Tracker is being considered. The team will investigate git blame features to ignore reformatting commits. A participant suggested "ruff" formatter as a potentially superior, faster, and more configurable alternative.
  • Technology Stack Upgrades: The Data Tracker will undergo upgrades from Python 3.9 to 3.12 and Django 4 to Django 5. A comprehensive review of its 53 first-order Python module dependencies will be conducted, followed by establishing a formal periodic review process.
• RFCs and Internet Drafts Not UTF-8 Encoded:
  • Discussion was held regarding RFCs and Internet Drafts that are not UTF-8 encoded, currently defaulting to Latin-1. This includes "encoding garbage" in some I-Ds and non-ASCII characters (e.g., names, smart quotes) in older RFCs, some of which are internationalization RFCs.
  • Community sentiment generally favored re-encoding these documents to UTF-8 for consistency and to simplify tooling, even if it meant changing bits in older documents. The need for IETF to "eat our own dog food" by correctly displaying characters on its websites was highlighted.
• Mail Services Update (New Machine Features):
  • The newly engaged "new machine futures" organization has improved mail system behavior.
  • Automation scripts for TLSA record maintenance are being enhanced to gracefully handle SSH server key changes. A synthetic probe is also being developed to actively verify TLSA records against served certificates.
  • Issues with localhost resolution during the migration to the mail2 host were resolved, restoring DKIM signatures for ietf.org. DKIM signing will extend to other primary domains soon.
  • Work is underway to move the Postorius fork into an open-source repository for transparent development of moderation features, such as providing direct links in moderation emails.
• Liaison Management Tooling:
  • A recent meeting with IB liaison managers refined requirements. It was confirmed that the existing "liaison contract role" is no longer needed.
  • High-priority requirements include direct control over the "from" field in emails (which is separable from a full application rewrite) and granting full access to liaison coordinators (which may require defining a new role).
• Cyber Security Services RFP: The evaluation of proposals for cyber security services is in its final stages, with an announcement expected very soon.
• FYI Notes:
  • NIST and IEEE BIB Entries: These entries remain stale because the IETF's system is using an older version of the Relaton crawler that is incompatible with recent changes in the raw feeds from these organizations. This issue will be addressed as part of the BIB service modernization plan.
  • Project Names "Purple" and "Red": These are project names for the RPC workflow tool ("Purple") and the RFC editor website ("Red"), originating from internal RPC cultural history (e.g., purple background for their workflow tool, "red" for RFC Editor).

Decisions and Action Items

• Decision: The TOOLS team will investigate and address the Data Tracker agenda materials popup issue to ensure materials open in a new window or tab.
• Decision: Re-encoding RFCs and Internet Drafts to UTF-8 will be pursued for consistency, ease of tooling, and correct display, with a focus on modern documents and clear encoding issues.
• Action Item: The TOOLS team will allocate parallel development resources to address pending feature requests from the ISG, IESG liaison management, and the cross-property search engine.
• Action Item: The TOOLS team will evaluate the ruff formatter as a potential alternative to black for Python code formatting.
• Action Item: The TOOLS team will continue improving TLSA record automation scripts and develop a synthetic probe for proactive monitoring, aiming for deployment by the next meeting.
• Action Item: The TOOLS team will prioritize implementing direct "from" field control and full coordinator access for liaison management tooling.
• Action Item: Robert will assess available data on automated traffic/crawlers for potential sharing in a MAP discussion.
• Action Item: The TOOLS team will conduct a thorough review of Azure managed database service access levels and capabilities based on community feedback.

Next Steps

• Continue critical development work on tools modernization, including the website, RPC workflow tool ("Purple"), and ARA system, in preparation for IETF 123.
• Advance the integration of Data Tracker and Mailman 3.
• Implement new strategies for managing automated traffic and for the periodic release of entries from block lists.
• Proceed with the separation of the authentication service using Authentic.
• Adjust the artifact storage model and decouple subservices for independent management.
• Begin the incremental migration of production services from Digital Ocean to Azure.
• Enhance backup strategies for improved resilience.
• Execute the planned Zulip 8 to 9 upgrade.
• Modernize the BIB service to address stale data and improve search capabilities.
• Implement technology stack upgrades for the Data Tracker (Python, Django, and dependencies).
• Await the announcement of the selected vendor for Cyber Security Services.
• The next TOOLS working group meeting is scheduled for the second Tuesday of May.