Session Date/Time: 09 Apr 2024 18:00

TOOLS

Summary

The TOOLS Working Group met to discuss the ongoing infrastructure transition, progress on key services like Mailman 3, DataTracker enhancements, and upcoming website updates. A significant portion of the discussion focused on the complexities of migrating the IETF's mail processing system to Amazon SES and measures being taken to ensure service reputation. Participants were also informed about the impact of the transition work on feature velocity and the expected return to normal development speed soon.

Key Discussion Points

Infrastructure Transition Update:
- Mailman 3 Transition: Still in progress, with the main transition expected shortly after April 12th due to ongoing complications.
- Services Successfully Transitioned: IETF website, Notes service, Bib, and Analytics have successfully moved since IETF 111. Follow-up work for future website deploys is in progress.
- Mail Relay: A central mail relay is established for new cloud applications, simplifying future mail system adjustments.
- Private Wikis: All private wikis, including a new empty instance for the IAB's private wiki, have been moved to DigitalOcean. Content migration will be manual.
- Older Web Services: Several older web services are being re-architected; for example, jabber.ietf.org now redirects to a Cloudflare blob store front-end with OIDC for potential access restrictions.
- Mail Processing Chain (Amazon SES): The largest current challenge is transitioning mail submission to Amazon's Simple Email Service (SES). Sirius is engaging Amazon for access and warming IPs to maintain reputation.
  - A participant raised concerns about mixed success with SES and suggested a backup plan.
  - Mark (Sirius lead) acknowledged the concerns, detailing precautions such as close monitoring (24/7), warming multiple IP addresses, and segregating potentially problematic mailing lists to separate IPs to protect IETF's reputation.
- Larger Applications Staging: Instances of the DataTracker and Mail Archive are expected in staging within the next two weeks to test infrastructure efficacy and gather performance feedback.
- Instrumentation and Status Page: Enhanced internal monitoring (Grafana) and a new external high-level status page (status.ietf.org) are in development to provide real-time service status, with plans for wider advertising once refined.
- Ticketing System: The Secretariat's primary ticketing system (RT5) is scheduled to move to a hosted service tomorrow.
- RPC Production Machine: The RFC editor.org origin (RPC production machine) is slated to move to DigitalOcean later this month, with application testing largely complete.
- Project Overview: Mark (Sirius) and Margaret (Painless Security) reinforced the complexity of the transition due to "archaeology and history" in the infrastructure, but expressed confidence in bringing it to production.
DataTracker Clarity Banner: A feature branch (DT clarity) adding a banner to non-IETF stream drafts/RFCs is almost ready. It will be updated, shaken out, and then merged to production.
Older Website Retirement: The process of turning down several older website instances is awaiting confirmation from Alexis regarding their capture by the Internet Archive.
Overall Velocity Impact: It was noted that bug fix and feature velocity on primary applications has been lower due to the focus on the infrastructure transition. This is expected to improve significantly within the next few weeks as the transition nears completion.
Website Update: A major update to the IETF website is scheduled for release tomorrow evening UTC. This includes a complete redo of the menu structure with "mega menus" for higher density and a more modern interface, along with underlying style changes.
Mail System Improvements: A future discussion will be convened with experts to address improvements to mail delivery, including application message signing (DMARC/DKIM) and domain configuration, to enhance deliverability across various mail providers.

Decisions and Action Items

Decision: The Mailman 3 transition is targeted for shortly after April 12th.
Decision: The DT Clarity banner feature will be merged into main and released into production once final checks are complete.
Decision: Older website instances will be retired and redirected to the Internet Archive upon confirmation of successful capture.
Decision: The Secretariat's primary ticketing system (RT5) will transition to a hosted service tomorrow.
Decision: The RPC production machine (RFC editor.org origin) will move to DigitalOcean later this month.
Decision: A major update to the IETF website, including new menu structure and styles, is scheduled to go live tomorrow evening UTC.
Action Item: Robert to organize a discussion with mail system experts (e.g., John, Taroh) to plan long-term improvements for mail deliverability and signing (DMARC/DKIM).

Next Steps

Continue close monitoring of the Amazon SES mail processing system (24/7 initially) to ensure reputation and performance.
Proceed with moving DataTracker and Mail Archive instances to staging for testing and performance evaluation.
Refine the external status page (status.ietf.org) and advertise it to the community.
Complete the remaining infrastructure transitions, including the RPC production machine.
Shift focus back to maintenance and feature development for primary applications (DataTracker, RPC modernization) following the infrastructure transition.