Markdown Version | Session Recording

Session Date/Time: 19 Nov 2024 19:00

TOOLS

Summary

This TOOLS Working Group session focused on a follow-up to the IETF 121 Data Tracker outage, an update to the tools roadmap with community input on prioritization, new observability tooling, the phased migration of the mail processing system, and options for making historical blue sheets available. Key discussions included architectural changes to improve Data Tracker resilience and performance, and the need for broader community engagement in tools prioritization.

Key Discussion Points

• IETF 121 Data Tracker Outage Follow-up
  • The root cause was identified as very poor performance of an NFS server, exacerbated by cumulative load since its last reboot and increasing automated traffic.
  • Mitigation strategies include:
    • Adding monitoring for NFS server performance.
    • Architectural shift to move primary access of artifacts (drafts, agendas, recordings) from file systems accessed by the Data Tracker to a blob store, allowing direct client access.
    • Improving heavy computation views by moving them to offline computation and populating caches.
    • Collaboration with Cloudflare experts to refine application firewall rules and rate limiting for better DDoS mitigation and load shedding.
    • The authentication system (auth.ietf.org) remained operational during the outage, demonstrating the benefit of separated cloud application concerns.
• Roadmap and Prioritization
  • The updated tools roadmap highlights major projects, primarily focusing on items in the leftmost column.
  • IAB Workshop Support: Clarified as enabling easier management of workshop materials, sessions, and associated papers, currently modeled as "teams" which is suboptimal. This is a medium-sized task.
  • Data Tracker History on People: A request was made to move this item (currently in the third column) to the middle column for earlier prioritization.
  • Liaison Tool Update: Strongly endorsed by IAB members as a critical need due to high volume of incoming liaison management. The proposed changes aim to simplify the system. Robert S. will link the requirements document to the relevant ticket and announce it on the tools-discuss list.
  • Community Feedback: Concern was raised about thin community representation in the call for prioritization feedback. Suggestions included posting notes to the tools-discuss list to encourage broader engagement.
• Observability (Grafana Tempo)
  • Plans to integrate Grafana Tempo into the monitoring stack were presented to add end-to-end tracing and observability points from browser to application to network.
  • This is expected to dramatically increase the ability to identify performance bottlenecks and focus optimization work.
  • The guides app is slated as an early test subject for Tempo integration within the next few weeks.
• Mail Processing System
  • A hosting service with good mail reputation, IPv4/IPv6 address space, and cost-effectiveness has been identified for IETFA's mail processing.
  • The migration will be a phased approach: an initial uplift of the existing system ("as is") to the new host, followed by a full-featured, test-driven rewrite of the mail processing system.
  • A backlog of feature requests for Mailman 3 is being collected, with a call for users to contribute to this list.
• Old Blue Sheets
  • The Secretariat has scanned historical blue sheets (dating back to meetings with a "2" in the first digit of the meeting number) and is planning how to make them available.
  • Three options were discussed:
    1. Integrating links into the existing, often handcrafted, HTML of older proceedings.
    2. Modeling earlier meetings in the Data Tracker with session objects to attach blue sheets.
    3. Creating a new, separate portal for accessing old blue sheets.
  • Option 1 (integrating into existing HTML) was acknowledged as potentially tedious but resulting in a static, manageable artifact. Option 3 was noted as simplest but with less discoverability. No strong preference was expressed, and further brainstorming was invited.
• Data Tracker Search Behavior Change
  • To mitigate the load from automated traffic performing search queries, the Data Tracker's search functionality will transition from GET requests with query strings to requiring a form POST.
  • This will involve obtaining a CSRF token and submitting a form, potentially adding an extra click for some users. Unintended side effects from internal Data Tracker links will be addressed if identified.
• IETF Trust Name Change (TLP 6)
  • Tooling updates (e.g., submission tool boilerplate) are ready to support the IETF Trust name change, awaiting the publication of TLP 6.
  • The publication of TLP 6 is delayed due to scheduling difficulties with the Copyright Contact Group (CCG).
• Tools Meeting Calendar
  • The WG discussed discontinuing the use of a separate Google Calendar for regular tools meetings.
  • Attendees were encouraged to rely solely on the Data Tracker subscription for tools meeting schedules to maintain a single source of truth.

Decisions and Action Items

• Prioritization Adjustment: A sense of those present indicated a willingness to swap the priority of the "IAB Workshop support" and "Liaison tool update" projects, with the latter considered more pressing due to active pain points.
• Liaison Tool Requirements: Robert S. will ensure the requirements document for the liaison management system is linked from the relevant Data Tracker ticket and will send an announcement to the tools-discuss list.
• Community Feedback on Roadmap: Robert S. will post the meeting notes (Markdown) to the tools-discuss list to encourage broader community feedback on roadmap prioritization.
• Tools Meeting Schedule: Rely on the Data Tracker subscription as the single source of truth for tools meetings; discontinue use of the Google Calendar for regularly scheduled meetings.

Next Steps

• Continue engagement on roadmap prioritization via the tools-discuss list and direct feedback.
• Proceed with integrating Grafana Tempo for observability, starting with the guides app.
• Begin the initial uplift phase of the mail processing system to the new hosting service.
• Further evaluate options for making historical blue sheets available to the community.
• Monitor for the publication of TLP 6 to enable the IETF Trust name change tooling updates.
• Individuals should subscribe to TOOLS meetings via the Data Tracker.