Session Date/Time: 13 Dec 2022 19:00

TOOLS

Summary

This TOOLS Working Group session focused on several key infrastructure updates and upcoming changes. Major topics included the planned migration of DNS services to Cloudflare, the increased involvement of contract DBAs for PostgreSQL, operating system upgrades for IETF servers, and the significant migration of the DataTracker backend from MySQL to PostgreSQL. Discussions also covered adjustments to consent pop-up mechanisms for IETF services and various ongoing development projects. A notable discussion arose regarding the strategy for addressing architectural issues in critical third-party tools that IETF services rely upon.

Key Discussion Points

DNS Transition to Cloudflare:
- The IETF DNS services are scheduled to transition from AMS infrastructure to Cloudflare in mid-to-late January.
- This will be a disruptive transition, likely requiring a period (approximately one day) without DNSSEC signing, a recommendation aligned with community input and Cloudflare practices.
- The plan is to continue leveraging volunteer secondary server infrastructure, though final configuration details are pending.
- An announcement will be sent to the community regarding the specific date and any expected disruption.
Database Administration with Sirius Open Source:
- Sirius Open Source has been contracted as virtual DBAs, with increased direct access to machines for PostgreSQL configuration.
- They are prototyping a more comprehensive backup and high-availability strategy on the sandbox environment, aiming for deployment on ietfa before month-end or early January.
- Challenges include OpenSUSE packaging for certain utilities like PG Loader, which will be addressed using Docker in the short term, with a move to stronger tools like PG Backrest planned for future infrastructure revisions.
- The number of individuals with root access to the ietfa server has increased, expanding the support team beyond historical reliance on a single individual.
IETF Server Operating System Upgrades:
- IETF servers are running OpenSUSE, and many non-production instances (sandbox, hot standbys) have already been upgraded to OpenSUSE 15.4 without issues.
- Production services on ietfa and ietfx are planned for upgrade during the first week of January, and ietf-a during the third week of January, after the year-end break.
- These are live upgrades, with individual services experiencing only single-digit second downtimes during cutover.
DataTracker Migration to PostgreSQL:
- The transition strategy in development environments has been stable for nearly a month, with high confidence in performance gains over MySQL.
- The migration will require a downtime estimated between 15 and 30 minutes, during which MySQL and PostgreSQL will be stopped, a MySQL snapshot taken, data migrated, and services restarted on PostgreSQL.
- The biggest current dependency is packaging PG Loader for the production environment, with a Docker-based solution being tested on the sandbox this week.
- Proposed Schedule: Thursday, January 26th, afternoon US time. This timing aims to minimize disruption following an IESG telechat and considering typical usage patterns.
- A participant suggested scheduling an interim meeting for the Tools team during the planned outage to clearly communicate the downtime, and this was taken as a good idea. Robert (Chair) confirmed that an announcement would be sent to ietf-announce weeks in advance. No objections were raised regarding the proposed date, indicating a sense of those present that this is a workable plan.
Consent Pop-ups for IETF Services:
- Following feedback from IETF 115, the practice of requiring consent pop-ups when using DataTracker credentials for services on ietf.org domains is being streamlined.
- For ietf.org domains (e.g., Medico sessions), consent for using DataTracker information is now assumed.
- For services not on an ietf.org domain (e.g., potential Gather.town integration, though not currently integrated), the consent pop-up will remain.
- Medico is already configured to use medico.ietf.org and has the consent click-through disabled. Yang Catalog, while an IETF activity, is not an ietf.org domain, but OIDC consent there only affects a small number of administrators.
FYI Updates (Brief Mentions):
- An RFI for infrastructure strategy is planned for early next year.
- Notes from the Tools Workshop on publishing documents are being constructed.
- IMAPD infrastructure is being revised to use an API to the DataTracker, reducing code coupling.
- IDNits and RFCdiff have been fully redirected to authortools, with a temporary option for RFCdiff output to address specific community feedback.
- Wiki.js deployment and CI/CD tooling for DataTracker and the website are progressing.
- DataTracker development shows high velocity, with successful deployment of timezone awareness and ongoing PostgreSQL migration.
- Yang Catalog: A discussion arose regarding the boundaries of contractor support for third-party tools like pyang, which Yang Catalog relies on. Michael questioned how architectural issues in such dependencies would be handled.
- BibXML and Miller Author Tools have open issues being tracked.
- Mail archive reporting via APIs is progressing.
- The Statement of Work (SOW) for Yang Catalog for the upcoming year needs to be defined.
Discussion on Third-Party Tool Support (e.g., pyang, libxml):
- Michael raised a concern about how the IETF handles architectural issues in critical open-source third-party tools (like pyang or libxml) that IETF services depend on, especially when community volunteers might not be available or able to make fundamental fixes without creating technical debt.
- Robert (Chair) indicated that the working plan involves using staff/contractors to provide guidance or develop frameworks to patch these tools as installed.
- Jay added that contractors (Nick and Kasara) are flexible and successful in tackling diverse problems, and mechanisms exist for specialist contract programmers for urgent, short-term fixes. The greater challenge is finding long-term developmental resources for architectural changes.
- Michael clarified his concern was specifically about larger architectural problems that require a longer view to avoid creating technical debt, which aligns with Jay's point about longer-term development.

Decisions and Action Items

DNS Transition:
- Decision: Proceed with planning the DNS transition to Cloudflare mid-to-late January, with an expected period of DNSSEC outage.
- Action Item: Robert (Chair) to send an announcement to the community well in advance of the transition.
DataTracker Migration to PostgreSQL:
- Decision: Tentatively schedule the DataTracker migration for Thursday, January 26th, afternoon US time.
- Action Item: Robert (Chair) to confirm the date and send an announcement to ietf-announce with weeks of notice.
- Action Item: Robert (Chair) to schedule an interim meeting for the TOOLS team to coincide with the outage, making it clear this is a planned outage notification.
Yang Catalog Statement of Work:
- Action Item: Eric to provide a summary of desired work and operational expectations for the Yang Catalog for the next year, prior to the Christmas shutdown.

Next Steps

Continue prototyping and deploying the new database backup and high-availability strategy.
Proceed with OpenSUSE 15.4 upgrades for production IETF servers in January.
Finalize and execute the DataTracker migration to PostgreSQL on January 26th.
Prepare for the RFI regarding infrastructure strategy early next year.
Review and address open issues for BibXML and Miller Author Tools.
Define the Statement of Work for Yang Catalog for the upcoming year.