Session Date/Time: 14 Jan 2025 19:00

TOOLS

Summary

The TOOLS working group meeting covered updates on several ongoing technical projects. Key discussions included the phased transition of email processing to a cloud-hosted server, refinement of requirements for the new liaison management tool, progress on integrating blob storage into the data tracker, and the use of Cloudflare managed challenges to mitigate bot traffic. Updates were also provided on the XML2RFC Trust updates and the roadmap for IESG dashboards.

Key Discussion Points

Email Processing Transition:
- IETF's email processing is being migrated to a cloud-hosted, openSUSE server, preserving its structure.
- The final production server has been provisioned, and its mail reputation is being established.
- DNS changes to include the new server's addresses for sending are pending.
- An IP warming program, gradually increasing mail volume from the new addresses, will start this week.
- The final cutover is tentatively targeted for the end of January, pending detailed planning.
- A brief downtime will be required to migrate Mailman databases, which will be announced to the community in advance.
- Related Change: Recent Postfix configuration changes on ietfa removed the reverse DNS requirement for incoming IPv6 mail and increased worker queue sizes, showing positive effects on machine behavior.
Liaison Management Tool Requirements:
- Requirements for replacing the liaison management tool have been converted into GitHub issues and discussions.
- These requirements are being vetted with the IAB and IESG.
- A specific discussion point concerns the "liaison contact" role in the data tracker. There is confusion about its purpose, a suggestion to remove it, and concern about its historical maintenance and potential impact on existing configurations. Community feedback is requested on whether to retain or remove this role.
Blob Storage Implementation:
- Progress has been made on design choices for blob storage integration.
- The architecture aims for the data tracker client to access most data directly from blob storage, minimizing data flow back through the data tracker application.
- Work is ongoing to mitigate latencies for processing paths that require repeated reads and computations from blob storage, likely involving caching strategies.
- New monitoring has been implemented for NFS mount performance to track potential issues if reliance on NFS continues.
Cloudflare Managed Challenges:
- Managed challenges are being used to slow down abusive bot traffic (e.g., fuzzers, high-rate scrapers) that overload the system.
- These challenges involve embedded JavaScript and user interaction checks.
- Currently deployed on the Data Tracker's login and account creation endpoints.
- Following an intense botnet attack from China during the holiday break, a managed challenge was temporarily implemented for most non-API endpoints for traffic originating from China.
- Feedback has been limited: one user reported challenges at the login endpoint but found a comfortable workaround. Another user from China reported being challenged on every request, despite the challenge satisfaction duration being set to a day, which is currently being diagnosed.
- A sense of those present indicates that similar attacks from other regions are anticipated, suggesting potential future expansion of managed challenges globally for most traffic.
- It was clarified that the Cloudflare challenge is distinct from the OIDC token expiration issues sometimes experienced between MeetEcho and notes.
- The challenge on login endpoints is to deter password cracking attempts. For API endpoints, credentialed access will be required for expensive operations if attacks occur.
XML2RFC Trust Updates:
- The implementation of XML2RFC Trust updates is currently stalled.
- While two of the three communities within the CCG (responsible for IANA protocol parameters) have agreed, one representative from the Names Community is strongly opposed, preventing consensus.
- The CCG has been given until the end of the month to resolve the internal disagreement before further action is considered.
IESG Dashboards:
- Work on IESG dashboards is currently prioritized behind the blob storage and RPC tools modernization projects.
- The tentative timeline for significant progress is March or later.
- It may slip past the Bangkok meeting (Madrid) if other major projects encounter problems, though critical parts might be cherry-picked.

Decisions and Action Items

Decision: Proceed with the email processing transition to a cloud-hosted server, targeting end of January for final cutover.
Decision: Cloudflare managed challenges are actively deployed to mitigate bot traffic; an additional, temporary challenge was deployed for most non-API traffic originating from China due to recent attacks.
Action Item: Wes Hardaker (or liaison coordinator Maria) to engage the IAB and IESG for vetting the liaison management tool requirements documented in GitHub.
Action Item: The community is requested to provide feedback on the "liaison contact" role in the data tracker via the GitHub discussions, specifically whether it should be retained or removed.
Action Item: Investigate the report of a user from China experiencing managed challenges on every request, despite the configured satisfaction duration.
Action Item: The CCG is tasked with resolving the internal disagreement regarding XML2RFC Trust updates by the end of the month.

Next Steps

Continue the email processing transition, including starting the IP warming program and planning the Mailman database migration downtime.
Further refine liaison management tool requirements based on IAB, IESG, and community feedback.
Advance the blob storage implementation, focusing on client direct access and latency mitigation strategies.
Monitor and tune the Cloudflare managed challenges, preparing for potential wider deployment if bot attacks escalate globally.
Await resolution from the CCG on the XML2RFC Trust updates.
Continue prioritizing blob storage and RPC tools modernization; IESG dashboards development will follow.