Session Date/Time: 24 Jul 2026 07:00
[00:00:36] Jean-Francois Tremblay: Hello everyone and welcome to the NMRG session. I'm Jean Francois. I'm Co Chair of NMRG with Jefferson Compost Norway. To start the meeting, of course, let me remind you the note one. Just to remind that IATF follows the IATF Intellectual Property Rights IPR disclosure rules. So it means that here you have all the details and the RFC, but basically if what you present will be covered, for example, by some patent, you have to disclose it. And just for information, today it will concern on presentations that has been our two disclosures for patent for the second presentation of today. And you can also look into that if you want into the APIs because your list that is on the data tracker. Although what is important as as reminding sorry. As reminding here is that we I mean, the RTF is our routine make recording of online in person meetings that includes audio, video, and photograph and previous recordings online. If you participate in person and choose not to wear a red do not photograph lanyard, then you consent to appear in such recording. And of course, if you speak at Microphone or appear on a panel or as a member of ITR, then you consent also to appear in your recording. And if you participate online and if you turn your camera or your microphone on, you also consent to appear in such recordings. Important is the privacy and code of conduct. Is additional to what I said before is there are also privacy policies that you mentioned here and that you can consult. And important is that we we all to work, especially with all our participants, in case of concern, you can contact the Ombudsman team. Although we have two here, three accessories that you can consult for more details. Now just remind that Nemert is a research group, it's part of the IETF, so Internet Research Task Force, in focus on longer term research issue related to the Internet, the partner organisation, the ITF focus on shorter term issues of engineering and standard making. So we are the ITF is not a standard development organisation. Still in research group, we can publish informational or experimental RFCs. But yes, just to remind you, we are a research group of RTF, so more focusing on research collaboration and long term issues. Okay. Some links for today that you are, I think, most of all aware. What is important that we mention here is that the recording will be available on the YouTube so you can easily find it as well. Of course, for the notes, also you can consult it after after the meetings as well. So I will not take too much time because we have, as you can see here, a dense so we have a different presentation. So I will just remind all the speakers to also take care of the time that includes questions. We may be a bit strict on time, as usual because we are a full session. So we'll put timer during your presentation, so please keep an eye on it. So thank you. I don't know, Jefferson, if you want to add something? Okay. So we can start with the first presentation. So Stefano, if you want to to come to the stage. So first presentation is Stefano Seci about Autosupervised Automation Loop System for Cloud Native Infrastructure.
[00:04:56] Stefano Secci: Hello everybody. Thank you Jerome for the invitation. Thank you the coaches as well. So I will present you the result of some year of research we did at LIP6 in Paris that has been taken over by a startup we have created, Nerotel dot ai.
[00:05:13] Jean-Francois Tremblay: So it
[00:05:17] Stefano Secci: is about an automation loop for cloud native infrastructure. So we have, let's say, defined our own way to implement the the MAPI system to create a closed loop automation of an infrastructure essentially composed of servers containers, radio front end. And I will present you today the result a little bit of this work. I will focus on the data pipeline system, which is there for the monitoring function, on the artificial intelligence functions, which is the core of the system, performs anomaly detection based on which we have a plan block which will take the anomaly fingerprint and based on that, the orchestrator. And this, of course, is an automation loop, so it will keep going. The principle is that, again, it's the same we know since the MAPE system was conceived. We want to go from the current state to a better state, a desirable state, every time in which we determine that the current state is deviating a little bit from the nominal or desirable conditions. So the challenge is therefore to determine when the state is no more normal or nominal or desirable. And this is the first challenge, so to detect the anomaly, qualify that. And based on this qualification, determine how to get back to the desirable state. We want to do that in an auto self supervised way. That is, we do not want to use labels, depend on labels and labeling processes. So let's we start with the data pipeline system, which is a very important brick of the system. Basically, we are in network infrastructure end to end. The reference use case is the five gs system, but it has been also tested with the IP multimedia cluster. That pipeline system is there to bring time series in real time from different possible data sources to the place or the places where this data can be exploited. So in the case of the five gs use case, it supports a number of endpoints, basically user equipment, containers and servers running, the run functions, disaggregated or not, and the five gs code. We use in our system a test bed, which is based on open source tags, so Open five gs, but also Open Air Interface, Qudu and other elements as well as commercial elements. Just for you to really understand what we are talking about, we as there are no public data set to test anomaly detection or reinforcement learning functions, we have built our own data set. We have published it. And to build this data set, we'll build a end to end test bed where we have injected traffic from real traces from a specific region in France. And by replaying this traffic, we have monitored the infrastructure and extracted the states. The state of the different nodes are basically covering CPU, memory, IO storage and the network matrix. The data set is made available. We are making different So you can go in the first link to have a unit. As it is pretty big, if you want to download it all, we will send a separate bulk link. And we are in the range of thousands of time series with a relatively high collection rate. We are in the order of one hundred and fifty to three hundred milliseconds for each of these thousands of time series. So this data is brought by a data pipeline system to artificial intelligence functions where we analyse it in real time. So this data streaming is done by a system that is basically a data pipeline system adapted from the IT world, where you have the data warehousing system with data lakes and the related system. We simplified it a little bit. We took only what we think we need in this system. In particular, we have two brokering functions that are following a Kafka system. They get the data from data sources. They balance them to the following functions. In the middle, have data processing, pre processing functions that are there to basically perform at least operations on the data, normalisation, transfer and put things in order to the data for being processed by processable by AI functions. So what do we mean with AI functions? We mean functions that can be distributed over the network or can also be centralised, to which we send this data real time coming from multiple sources. We are in the order here of thousands of sources, if we think to an operator network, even hundreds of thousands for a large scale, nationwide operator, passing through these different data processing functions. So the input brokers that are able to balance and aggregate to data processing functions, which in turn can give the result of data processing to other egress brokers that in turn can do the load balancing toward one or multiple artificial intelligence functions. So let's have a look at these artificial intelligence functions that we named in our European project AIF. So again, we wanted to have a system that is auto supervised, self supervised, that doesn't need labeling. So in order to detect anomaly, we do not want to associate a given state of the infrastructure to a known anomaly. But we want to define freely what an anomaly is, give a fingerprint of the anomaly, and thanks to this fingerprint, let the the automation system define how to recover that. So and we are, therefore, using a relatively standard anomaly detection framework based on an encoder decoder architecture, typically LSTM, but can be also other encoder decoder architecture. The intuition is the same. We have an image of the infrastructure by means of these thousands of time series. We want to train the network so that the output will look like the input during the training phase. So this training is therefore done by minimising the reconstruction error so that given all the inputs, this reconstruction error over the training phase is minimized. Of course, the challenge is to give in the input the states that are really considered as normal as there is no perfectly normal state. There is here a big challenge in determining how to select the input to train the network. And in the inference step, go then checking the current state based on the trained network. And if the reconstruction error goes over a certain threshold, then you spot an anomaly. This threshold can be computed in a custom way. The way people typically do, it's based on statistical thresholds. So for example, you will take the ninety ninth percentile of the aggregate reconstruction errors. You can see the reconstruction error here. The J are the different metrics. Or you can even group metrics among them so as to get a reconstruction error, which is the sort of average of the reconstruction error of the different groups. So that's relatively straight. The thing is that while we could live in collaboration with Orange because we had a PhD with Orange. While we could live with a single AIF for the IP multimedia subsystem, we could not do that while passing to the five gs system, which has much higher number of time series. So we went for a relatively classically here again federated learning based distribution of the artificial intelligence function so that we can have edge IIFs to which we have to bring the data in real time, which are then training locally their own network. And based on this local training, they have aggregation with a centralised or semi centralised. There can be multiple levels, servers that will then distribute the updates. This artificial intelligence functions can therefore be distributed. It has therefore an interface to get the data from the data pipelining, an interface to data potentially with the other artificial intelligence functions. In the case of federated learning, this data plane horizontal interface is not used, but it's the control plane interface which is used to exchange the weights with the other AIFs and then you have an outbound interface. The nice thing here that we enjoyed a little bit to do is to model the fact that the management of the struggles that is you in order to have the highest possible quality learning, you need the data to come in a semi synchronous way, in a synchronous or semi synchronous federated learning. But if by chance the data arrives too late, then you may lose the occasion to use this data for the learning. So you have so called stragglers. So we model the latency in this system as being the sum of the latency to get the data from the data sources, the latency to have the aggregation with the federated learning service, as well as the training latency at the AGIF. So you have an end to end latency which is mixing propagation delays, transmission delays and the training delays. And of course, you would like to have the variance of this latency at the different AGIFs, which is minimized. So how you can do that? By enabling AIFs in the right places and or by enabling on demand other accelerators to accelerate the training latency when the propagation delay is too high so that the training can be reduced. So this is a small game, as you can see, of balance between different delay components. And then so this AIF will end up by spotting anomalies. So if you allow inference at the edge, you will also have anomaly detection functions at the edge. Then when the anomaly is detected, you will require a way to characterise these anomalies. So a fingerprint, as I was saying. And thanks to this fingerprint, you will try to find a way to recover from this anomaly. So again, we want it to be self supervised. So you have to imagine this fingerprint as an array of reconstruction errors that can either be at the feature level, so thousands of reconstruction errors. So you have a big array and that is the fingerprint. Some of these reconstruction errors will not be anomalous, but a subset of them, in case an anomaly is detected, will be above their own threshold. You can also aggregate these thousands of features at the resource level. So you could say that all the metrics related to CPU states, for example, in standard containers, CPU states are characterized by 50 to 100 different metrics, including temperature, page memory, faults and so on and so forth. So you can also group by feature. This is what the theory suggests that has been demonstrated in AI literature that metrics that are strongly correlated with each other better group them together and perform the encoding and decoding functions at this group level. So this is what we did. In fact, we grouped the CPU memory network disk group, GenoDB. And for this part is not yet really developed, but we could also have a dedicated group to states related to user equipment or to other type of access networks, such as Wi Fi access network. And you can have these at the container level, but also at the physical server level, so that you can monitor possible propagation. And in addition to this group, we also work to have network function level, a specific matrix. So for example, by integrating 3GPP KPIs for the five gs core. So out of these, again, big arrays, big or very big arrays of reconstruction errors that form the anomaly fingerprint. Each element of this array is a sort of dimension. So you are navigating in what I try to represent here, a multidimensional space of points, infinite number of points, where an anomaly that you spot is one point in this multidimensional space. Here is, for example, you have only three dimensions, the reality of thousands. So that when you spot an anomaly, you are in a certain point in this multidimensional space. And, of course, your objective is to get back to the normal condition, To get back to the convex hull or the convex hulls, different regions that you know to be normal. You know to be because you did the training against them. Okay? So the challenge is how can we work in this space and have a mechanism that is able to automatically find how to go back. Knowing that how to go back implies applying actions such as scaling, rerouting, load balancing, bucket duplication, stuff like that, so that you go back to normal conditions. So this is a relatively natural field for our reinforcement learning, deep reinforcement learning, and the like where the actions can be defined in a dictionary of actions that you define, that you can, in fact, personalize based on the region where you are. So if you are you suppose that you spot an anomaly that is in this multidimensional space extremely far away And moreover, it's not close to any anomaly that you visited in the past. So we'll have really a potential, a very long list of reconfiguration actions to perform. On the other way, if you spot an anomaly, which is, from a Euclidean's perspective, extremely close to an already known anomalies, seen anomalies, you may have the intuition that you could reuse the actions that you already used for that to mitigate that anomaly. Okay? So that you could compose a set of reconfiguration actions with a certain confidence on your capability to get back to normal conditions. So that's it. So it was a very synthetic way, I wanted to convey the intuitions behind this framework, which is currently based on this anomaly detector that I proposed to you that gives the fingerprints of the anomalies to the reinforcement machine learning agent that has its logic to find the set of actions to implement in succession or by iteration. And we are currently thinking about replacing these anomaly detection deep reinforcement learning block with a word model, which is supposed to get rid potentially of the reinforcement learning so that you directly aspire the DPS, the data pipeline system, MatrixLIVE and give as a result the actions immediately. We follow the two steps, which are currently a bit challenging, not impossible to implement as we did, but which may be not extremely robust in case of very unstable states, fluctuations, state that change too fast, for critical systems. That's it for today, if you have a question.
[00:22:48] Jean-Francois Tremblay: Thank you, Stefano. Any question? Yes, we have John.
[00:22:55] John: Please. It's John-Fritz Question on slide 13, more or less for clarification. I understand that you're using a lot of network elements and data, but are you also using the UE data for the
[00:23:11] Stefano Secci: So for the moment, for for the UE, I put it there because it's something we would like to do for some UEs, vehicular UEs in particular, where you have the hands on the UEs for vehicular UEs, for example, of the train or over the European project also over the plane or the cars, where The UAE is not a personal UAE, but is an infrastructure. What we have anyway from The UAE is the radio front end matrix that SRS and operating interface, they exposing are a number of KPIs on a UE level that we are collecting them. Yes.
[00:23:47] John: A closely related question. There's a function called NWDAF that's defined in 3GPP. Are you using any of those concepts in there, or this is completely separate from
[00:24:00] Stefano Secci: In WDAF. That one? In 3GPP, five g core? This is NWDAF. Yeah. So network analytic functions is there natively meant and defined to process data that are coming from five gs core functions. Basically, the KPIs from defined by 3GPP. We go a little bit beyond that, even if also the NWDAF, as far as I know, I'm not really following 3GPP, but as far as I know, 3GPP is also a thing to extend the data scope of NWDAF to cover other data than KPIs from the five gs core functions. So there can be an overlap. There can be an overlap even if we are, again, scrapping the data from the container level, so which goes beyond the KPIs of the five gs core functions, the physical servers and also very different ends. Thank you very much. Very interesting. You're welcome.
[00:24:53] Jean-Francois Tremblay: Thank you, Stefano, again. And we move to the next speaker now.
[00:25:02] Peter Boers: Oh, where is my printer? So
[00:25:11] Jean-Francois Tremblay: next speaker is Peter. Just stop this one.
[00:25:22] Peter Boers: I
[00:25:29] Jean-Francois Tremblay: have to stop first, I think.
[00:25:34] Peter Boers: Stop it.
[00:25:37] Chengsu: Can I also directly share this screen? No?
[00:25:40] Jean-Francois Tremblay: No. No. You have to stop.
[00:25:48] Christophe: Sorry. I'm just
[00:26:01] Jean-Francois Tremblay: Every time I look no. I have to grant you
[00:26:05] Peter Boers: should be okay. Yeah. Alright. Good morning everyone. No, I think I'll keep it here. Thank you. Good morning everyone. So nice to meet you all. My name is Peter Boers. I work at SURF, the Dutch National Research Education Network. And in my function, I'm the tech lead of the automation orchestration framework, and I was asked recently to come present here. So I'm very happy to be here. So first, a little bit about SURF. In case you are not aware, SURF is a Dutch national research education network. We provide all sorts of IT services to higher education in The Netherlands. And the networking department that I work for is responsible for the fiber infrastructure and layer two, layer three services that we provide to our customers. All services on our network are fully orchestrated and automated, so there is no manual config. This is all done by our homegrown system. And, yeah, if you think about our scale, so in The Netherlands, we are quite large. Netherlands is a small country, but for The Netherlands, we are a large organization. We transport quite a lot of network traffic. And, yeah, last year, we transported about 1.6 exabytes of data, which is three ninety two million DVDs or 1,100,000,000,000 floppy disks. And this picture I generated, but it gives you a good idea of why NRENs exist, because NRENs are in the world so that they enable science, so that they enable scientists to move data from their experiment to compute facilities. And yeah, if you imagine that people used to fly around the world with hard disks or floppy disks to move data, the reason why NRENS exists is so that doesn't happen anymore. So a bit about the work that we are doing in the area of intelligent networking. So our ambition is to have an intelligent network capable of correctly assessing incidents and fixing them in real time with AIOps. Right? So that's the bar that we've set for ourselves. And yeah, you know, that's a difficult problem. And you need to break it down and try and figure out how you can achieve that. AIOps, I'm sure you are aware of, but this is sort of our mental model. On one hand, you have data coming in. You observe that data. You do something with it with AI and machine learning, and then you can have a reconciliation loop that acts on the anomalies or the things that you detect. So in our stack, these are the basic building blocks. And today I'm going to talk about Sherlock and Clippy. I think that's sort of the building blocks that we have. And if you look at our stack and what that sort of looks like in practice is that we have a telemetry pipeline that is ingesting around about 250,000 events per second on the network at a resolution of about one event per fifteen seconds per sensor that we have defined. That gets sanitized through a streaming database, and eventually, that sanitization narrows the amount of events down to a reasonable amount that you can process with an inferencing system. And that inferencing system is is is Sherlock. And then once that inferencing system has done its work and either detected anomaly, yes or no, it will be passed on to Clippy, and Clippy will suggest something. And at the end of the day, that suggestion will either be approved by a human or not, and then our orchestration system will then pick up that and then remediate the problem. That's sort of the basic principle. You see here all the colors from the previous slide. You have the observing, you have the engaging, and then the action on the network. So yeah. Defining the use case. So you need to start somewhere. Right? And when we started two years ago, we decided to focus on highly disruptive events and incidents that are very visible in telemetry, but are not a binary event. So it's not up down. It's something that you want to be able to detect that has a little bit more nuance. And for us, because about 50% of our traffic is commodity Internet, that was a very good use case to start with, because it is something that you can probably model quite well. Yeah. And this is our first iteration of this guy. So what we did is we defined a couple of sort of high level goals that we wanted to achieve. The first thing is that we wanted to have a system that is highly scalable, because we need to we have quite a number of external peers that we need to model, that we need to monitor, needed to be easy to train and easy to run. And we came up with the sort of the mental model of how you could look at an agent system that could model what the Internet looks like from our perspective. So you can see that in the diagram. There's sort of like a VRF, then there's peers, and each peer has a number of neighbors, and each component will Would then have a machine learning model that would monitor that and help detect anomalies. This is our first iteration. Yeah. So what we also said is we wanted to evaluate a number of models, LSTM and transformers. Those two models, they could be useful, and they might be a good way to see if we can figure out if we can create something that can accurately model our network. Yeah. And basically, can we detect this? So this is a graph of a traffic with one of our peers, in this case, Microsoft, and what see here is that there's sort of like an anomaly that happened, and it existed for quite some time, and yeah, basically it was an inefficiency in the routing of our network, and we want to be able to detect this type of stuff. Right. So Yeah. Our first version took quite some time. I think about 90% of the time was basically an integration exercise, where we tried to figure out how we could deploy something in a production system where you have a battery of inference doing all sorts of stuff, and trying to monitor all of our external connectivity. And if you look at the white diagram, you can sort of see what we came up with, because LSTM was the model of choice in our first version, is that we had a system which would sort of retrain our LSTM models every so often, deploy them inside of our data pipeline so they could do the inference, and then, yeah, we could detect anomalies. Right? So you can see that as well. Which sort of showed that the technology might actually work. And then, yeah, moving on. So these are sort of the results that we had on the LSTM model. And for those of you unaware of how LSTM works, basically, the way it works, it tries to infer the next point. So the example here is the cat sat on the dot dot dot, and it will know, due to its training, that it probably sat on the mat. Right? LSTM was reasonable, right? It worked well. Transformers, our first version, wasn't that good. And for transformer, this is obviously the T in chat GPT, right? It works differently, has different algorithm, and it knows the relation between different types of things that it sees. Which intuitively sort of is something that's very interesting, because you might be able to leverage that if you want to correlate stuff between data points from a node. Transformers are way more performant. Right? So it it is faster. It scales better because you can create one model to rule them all, so to speak, instead of having a battery of LSTMs that you need to retrain every so often. Transformer is capable of being trained once, maybe fine tuned every so often, in the order of maybe once or twice per year. Whereas LSTM, what we saw with the way that we did it was you can train it really, really fast, but you need to retrain it quite often. That's of the experience that we had. But as you can see from this diagram, our first iteration of transformers was not that good. Yeah. So once we've done the integration exercise and once we've sort of analyzed how stuff could work, we decided that transformer was probably the way to go, but we needed to change the architecture of the model just slightly. And I have to really give credit to the people at our supercomputing center in The Netherlands who really helped me with this. We chose to move to inverter transformer. And our architecture changed quite significantly. So instead of having a streaming pipeline that would monitor every single event, we and that process would do the inference at the same time, we sort of pivoted to an Oracle endpoint. So basically an HTTP endpoint that you can hit with a tuple, and then get an inference result. And if you look at the model that we are running at this point in time, it's one parameters, 100 and these are sort of the vital statistics, and it's capable of inference, doing an inference with a step time of fifteen seconds, and predicting 32 steps ahead. And yeah, it takes about four hours to train on a on a on a very beefy GPU. And this is just a diagram. It's very difficult to see, but it's more to give the give the rigorous the amount of rigor that we put into this effort, is you can see all the training runs that we did, and the model that we actually selected is the one that sort of looks like this, when you look at the loss over time. So on the y axis, you can see the steps that you can predict ahead. And obviously step zero is very easy to predict, and step 31 is a little bit more difficult to predict. And then, yeah, the bluer, the tint, is the better performance that you get. And the loss after the 81 evaluation steps is around, well, 0.3, something that Or 0.35, something in that order of magnitude. And there's another graph showing the same, basically, where you can sort of see the loss over time. Which, yeah, sort of it does indicate that it trains well. At some point, it sort of stabilizes, it does continue to go down. Yeah. So Deployment setup. This the deployment setup. We have a stream processor that is able to sort of buffer everything and to make sure that we get data in order and on time. And every fifteen seconds for every BGP neighbor that we have on our network, we do an inference post to that machine learning model. That telemetry is also stored, and the result of the inference step is also stored. And then we can visualize that in Grafana to see how our model performs. And these are sort of some of the graphs that we have. Also very difficult to see, but you can sort of see the inferred
[00:40:29] Jean-Francois Tremblay: traffic.
[00:40:32] Peter Boers: So that's interesting. Do we see traffic at the same at the performance that we expect inferred in, inferred out? But you can also sort of see the thresholds and whether it can detect an anomaly, or yes or no. And you can see a small red bar. In this case, it detected an anomaly at a certain point in time. And whether this is a false positive, yes or no, we're still trying to figure this out, we need to fine tune the model, yes or no. You can sort of see it actually working. Right? And, yeah, this is a very exciting step for us, that it actually worked in production. But wait. Can't you just use moving average? Right? Especially if you're just looking at traffic. And well, yes, you could. But the thing is, due to the architecture, and especially if you look at the Inverted Transformer architecture, you can see you have t is time, and a, b, c are sort of the different metrics that you are interested in. And as you can see with Inverted Transformer, you get attention between different variables. That's very interesting, because if you run this, then you can sort of start detecting anomalies that come across from different variables, so you can start detecting if things have influence on one metric or the other. And it's also easy to scale, right? Because when you fine tune this model, you can easily add metric d, c, d, e, f, g, etcetera, without much effort. And you can start seeing attention across different metrics. So for us, even though it's in the first situation, you just see can we infer traffic, we also infer other things. It's just a thing that In the plots that I showed, we just express the traffic. So our model is actually capable of inferring quite a lot of metrics at this point in time. So conclusions about Sherlock architecture that can be extended, scalable, still needs some work, and it might actually be too good in the case, in the sense that it is able to detect patterns too easily. So you actually want a model that infers a little bit That the inference is more stable, but it's actually able to infer able to infer too easily in the sense that it changes its pattern too fast. And this is probably due to the fact that in the training set, there might be actually too many anomalies that it actually can follow. So rounding off, think. What about Clippy? So Clippy is other platform. It's co funded by EU and the FNS. It's basically an agent to agent architecture that is capable of doing autonomous troubleshooting. And we've recently done a demo where we did that across domains. Right? So we had our own agent to agent setup that was capable of talking to another agent to agent setup across The Atlantic with our partner organization, Internet two. Tying this all together, you can imagine that if we are able to get a machine learning pipeline and inference set up that is capable of flagging anomalies correctly, we can start an autonomous troubleshooting in this setup, and maybe even get that running across domains. Yeah. This is a screenshot of that actually worked. And I think I'm going to leave it at there. So this is our our our road map, and eventually, we're probably going to rename Clippy to Watson. So Thank you very much.
[00:44:59] Jean-Francois Tremblay: Thank you, Peter. So we have time for a few questions.
[00:45:06] Frank: Hi, this is Frank. Interesting work. How does the model that you train compare to many other time series forecasting models like Times FM, Moirai, Kronos, Cisco's Time Series Foundational Model, TODO and the likes, right? So I think this is something that I think MiXLA kicked off some three years ago to use transformers for predicting, and they're very good at that. We know that. So I just wonder how what you're doing compares to what the industry is doing.
[00:45:42] Peter Boers: Good question. Full disclosure, I'm not the guy choosing algorithms. But I know we are going to be evaluating TOTO in that sense.
[00:46:01] Frank: Yeah, would be good to run your model against public benchmarks in that space. So there's public benchmarks that are specifically done for benchmarking against networking cases. So it would be good to go and see that.
[00:46:17] Thomas Graf: Thomas Kauf speaking. Can you go to Slide 11 briefly where we see the network traffic and basically the routing change?
[00:46:29] Peter Boers: Slide 11. You have to help me. You have to say stop when you see it. Okay. So I think it's probably This one. Exactly. Yeah.
[00:46:48] Thomas Graf: So you were mentioning LSTM.
[00:46:51] Peter Boers: Yep. How about seasonality? Yep. So the LSTM model had the notion. So we we gave it some some information about time and about about time of day day of the week. Yeah. The thing is we needed to retrain it quite often to keep in touch with seasonality. So we had to retrain it every two weeks at least.
[00:47:15] Thomas Graf: Okay. But you have, like, seasonal event. Like, we just had sports events, like, a couple of weeks ago. So where I'm going to is, like, here, you have a routing event. Right? Yeah. And I think you're looking at the BGP neighbor, if I'm not mistaken. Right? Have you considered also taking other network events into considerations, like routing table changes and Yeah.
[00:47:39] Peter Boers: Sure. I mean, yes, Obviously. It's very difficult to deploy this at scale and get that streaming pipeline and the data integration set up. But, yes. So I've showed all diagrams just about traffic, but there is much more in the model than just traffic.
[00:47:59] Thomas Graf: Mhmm. Right? So, yeah. So just one recommendation. Maybe have a look at BGP monitoring protocol. Yeah. Where you natively get that out. Yes. We have not. And also think about the transformations you're doing afterwards and taking basically the taxonomy the relationships into consideration.
[00:48:15] Peter Boers: Yeah. Mean, I can't convey the complexity of the model, but yes, those things are in there. Definitely Yeah. Interesting work. Thanks. Yep. Thank you. Yep.
[00:48:26] Diego Lopez: Just two questions. One is quite simple. You are not using the data sets are not level.
[00:48:33] Peter Boers: Right? The data sets are not?
[00:48:34] Diego Lopez: Are not level. Not associated with a particular event or a particular you mentioned that you're considering going for labeled datasets is for further work. Right?
[00:48:45] Peter Boers: Yeah. So the the datasets is is it's just streaming telemetry and some syslog and BMP information is in there. That's that's sort of that's sort of about it.
[00:48:56] Diego Lopez: And the second is just an idea, probably more for the, I don't know, for for the project. Have you considered to use this as some kind of emulator of a of a BGP neighbor just to
[00:49:10] Peter Boers: Yeah. So the goal is that once we have this set up, that we should be able to have a model if we add more layers and more complexity that is is capable of being sort of like a digital twin. That's the ambition.
[00:49:23] Diego Lopez: No. Exactly. Well, it's a Yep. When when you have something like that, it would be very good that you can be able to tell us anyway. Thank you.
[00:49:31] Chengsu: Just a last question. Leandro?
[00:49:35] Peter Boers: Well, one, after I will look for you. How often do you retain this model for a year? So so we have two and a half years of data. And at this point in time, we've trained it on four months of data, and we haven't had to retrain it yet. We expect that we have to retrain it once every so often, but we're not sure yet. We've just deployed this a couple months ago, and it's still running fine. So we haven't had to retrain it yet. And the time to train? Is Four hours on that. Four hours off. Yeah. Four hours on the h 100. So, know, you need some serious infrastructure.
[00:50:18] Jean-Francois Tremblay: Beer?
[00:50:20] Peter Boers: So For each beer? Yeah. It's just the whole network. Okay. So we take we take data from the whole network. We put it into a model. It runs for four hours, and then we get this, and we can deploy it. And you don't need a so we we do need because the amount of events coming in per second, we do need a GPU for the inference. But you just need a tiny GPU. It doesn't need to be anything special. Yep. Thank you. Thank you. Alright.
[00:50:47] Jean-Francois Tremblay: Thank you again. So let's move to the next presenter. So Christophe Varianz. So here we have three presentations. Me stop this one.
[00:51:04] Christophe: K. I'll try not to make you listen to me for thirty whole minutes. The third one will be quite quick. We might spend a little more time on the first. These are this represents content from two later breaking, drafts that I started the ideas with Naji and Diego, last year. And some of the later breaking stuff in this companion drafts, is based on conversation with Brett having looked at the first. So I wanna run you through the ideas. Is there a clicker
[00:51:36] Jean-Francois Tremblay: at Oh, here's one.
[00:51:38] Stefano Secci: Thank you. Just one second. Yes. Sure.
[00:51:44] Christophe: This one okay. So the the the context here is okay. We love to do Yang models in IETF. Why do we do them? Because when two systems want to talk or understand information, let's say communicate, they need an unambiguous basis on which to do so. So they need to agree on what entities are part of the universe they're going to talk about. They need to, agree on what they're going to be called, how ideas are represented, and then a sort of grammar. Right? You get all three by fixing a data model. But, of course, human beings don't require this sort of, agreement in advance. Right? They can work it out on the fly. If you go back to the Bronze Age, a Phoenician trader could land on an Egyptian port, and they could figure out how to work out a deal to sell copper. Right? So this tells you a few things. One is that why are we endlessly generating Yang. One of the reasons is that they're brittle in a sense. They're they're very conversational purpose specific. Right? There's by definition, you never have the right ontology for every conversation unless it's an unlimited one, etcetera, etcetera. Right? So the idea is when we have cognitive AI, cognitive agents that can, in principle, communicate intelligently, reason, plan, and learn, can we approach things differently? You still need unambiguous interpret unambiguous interpretation of of communications, the semantics. But can they work it out more for themselves? And does that have implications for the data structures that are, worth standardizing or or that serve as references? So the the general context here, and there's many detailed use cases you could work out around the theme, but the basic idea is we have two systems that want to communicate and they're both cognitive. Something on you know, in the structure of this, I wanna focus on lexicon, which is a really a a a piece of an ontology. It's what the entities you're going to refer to are called or attributes or predicates, etcetera. But it's basically nomenclature, which is a subset of ontology. A data model is all of that in a specific sense represented in a specific grammar. Right? So we're going to sort of what we've discovered is that the thing that's really important here is actually lexicon, because it's where you kind of you need to start a mediation or reconciliation process. It's where most of the work goes. The rest is a little bit easier if you get that right. And it kind of suggests that almost from the start that reference lexicons rather than, reference ontologies, which are more flexible than data models, but still they're they're specific. Right? So they're less you know, lexicon is far thinner and and less brittle than than either of those two. But you wanna think about it as potentially more than just names. Right? And there's some you know, what's going to assist in this ambiguation. So in the flow of the the the demo here, it's, again, many possible use cases. This one is two agents that are they they're using different data models, which happen to be standard defined, and they want to talk to each other and agree on how to translate between the two, right, with no humans involved in that process. So think of it as like an automated system integration, task. So first of all, you know, they're passing messages over an eight to a bus. They agree on a process. Right? And the process they agree on is basically we're going to sort of lift our data models to ontologies. We're going to sort of try and map or represent the lexica that those represent. We're going to agree on those. After that, we can identify ontological gaps in detail, abstraction, or granularity. For example, in scope, what is within or not within a specific ontology. You can add to the one that has the the deficient ontology. And then, you know, how are rep how are differences in grammar, if you like, structure, represented ontologically. And they do this by conversation. Right? So starting with okay. Let's look at the lexicon. Let's try and do, simple matching. Recognizing this this isn't done, you can get false cognates, but you get a good starting point. Then you've got to do another pass which starts to draw another information, what what this is calling the gloss, which is that sort of explicative, ampliative, disambiguation supporting, information that you can draw on and eventually get through the process. Having done that, you've already got a got a good handle on where the, detail and scope gaps may also lie. And we found that the machines are actually very good at sorting through those on their own. And in this process, what you end up with is, you know, first of all, you've done this sort of generated translation artifacts for the, for the data schema. Then you can get on to, you know, the specific instance naming and things like that. Right? When you have specific knowledge graphs and not just generic, ontologies, we agree on what a table is, but I wanna know that, you know, I call that table one. And when you call it table seven, you're referring to the same thing. But there's a a process for that that works. And eventually, you can sort of validate the artifacts. Right? Depending on the the problem, you may be able to do things like, hey. If I wanna push a change to the network, I represent that in my data model. It has to go through a full round trip double translation, so I'm satisfied I get the same artifact back. I know we're both going to do the same thing, for example. If you can have reference lexica or at least having done some of this heavy lifting, post to the public domain, some of these reconciliations of reference lexica, you've taken out a lot of the work, and this becomes a lot more scalable when more systems are are trying to do this. The implication of all this is, you know, I think we validated, and ours is not the only the only work. This is an active field. That cognitive agents can can work out a lot of things in order to communicate and represent information unambiguously, much more flexibly and with much less sort of human created or standard support than we are used to in the past with noncognitive systems. The the go forward topic we're looking at is, you know, how to support this. Okay? If you can sort of step away and maybe you don't step away from, you know, Yang Yang data models entirely. They're excellent references. Depending on the case, you may need to build from them. But, you know, if it's Lexica, which is the more flexible tool, but it's not just a simple Lexicon, what are the things you would need to augment it with in order to serve in this kind of environment as referenceable objects while minimizing the scope for problems or the cognition workload in reaching agreement? Right? And there's some examples here. Okay. That's it for that one.
[00:59:52] Jean-Francois Tremblay: You want to take question now?
[00:59:54] Christophe: I I can. Yep. Sure. Any questions?
[01:00:03] Ken Watson: Hi. Ken Watson. When you're talking about agent to agent, I mean, it makes it sound more like peer to peer. But most of the time, I hear about agent to agent as more like, you know, top down. Like, the orchestrator talking to a controller, controller talking to the elements. So it's more like the scope of, you know, the control.
[01:00:25] Christophe: Well, I mean, we we've taken it simply as a protocol for communication between two agents regardless of their
[01:00:33] Ken Watson: Well, I mean, you talked about table one. I call it table one. You call it table seven. What's the actual practical network application of that example?
[01:00:42] Christophe: Well, in this case, we have two distinct you know, we have different data models on each side. Who is the different datas data model schema?
[01:00:51] Ken Watson: Who is we? I'm sorry? Who is we in your example?
[01:00:55] Christophe: Okay. Well, sorry. I thought I explained the use case, but we have one network. We have two independent agents which are using two different data models to represent that network for configuration or other purposes. Right? And they want to be able to talk to each other. Right? Or whether, you know yeah. Sorry. Diego, may maybe maybe you want to intervene.
[01:01:19] Diego Lopez: Just to try to provide an example from the some experiments that we have running. Think about, well, public network operator and a private network in a corporation that they are to they have something, some smart stuff there that that has to communicate and not necessarily are sharing the same ontology and they're not sending the the same and these are more peer to peer relationship. For sure, the the public network operator will be typically bigger, but at the end, you have to put in. This is one thing. The other thing is within organizations like mine in which people that are running the radio don't talk very often to the people who are running the fiber, for example. That's another case.
[01:02:01] Christophe: Yeah. I mean, I think, like, a like a lot of AI things, we're looking to automate things that currently have to involve people because they involve planning, judgment, learning. Right? We now have technologies that can do that. How can we apply them in some of these situations where we've had to rely heavily on standards to define information or data models, content structures? That's the that's the general problem statement. Okay. The next topic, little bit, different. This one was motivated by a discussion some participants in at CCAMP where topics of closed loop and intent conflicts and their coordination, got a lot of, attention. It's also a very active research field, and we connected with some some people doing research in that area. It was it was motivated by an NMOP draft that, I'll refer to at the end. But it's a so this is basically a a survey of a complex, you know, where the what we know about this, topic. Because there are things it's a bigger topic than you might think, more complex and potentially difficult and important topic than than you might think. And it may have implications when we think about network management architectures with agents in them, beside them, around them, on top of them, whatever it is. The main thing to understand is that because the whole point of agents is that they're supposed to act independently, the decisions they take and ultimately the actions they try to drive can conflict with one another. So what do you do about that? If they're just sort of generating information that, you know, it's information, then you don't have to worry about that. But if that information is leading to action decisions, then you do have to worry about it. Right? Then you have an issue. And the conflicts and coordination that is how you detect, preempt, or resolve those conflicts are flip sides of the the same coin. Again, it's it's not you know, an agent which merely generates some information to be digested by something is not going to be a problem. An agent which tries to drive action is something you need to pay attention to because it's at the actions or the decisions leading to the action that that the conflicts arise. And there's a whole bunch of different types of them. Right? Some are hard. That is, you ask me to do this, something else asks me to do this, and the two are in hard conflict. More often, they are soft. That is, you know, you want a lot of bandwidth, but you want it chiefly. Well, those don't tend to go in the same direction, but whether it's an actual conflict is a matter of degree and circumstance. Action ordering and impact. Right? I mean, two things, actions that look independent, but once you get the result of both or maybe dependent on the order in which they happened, you've got a result that that neither liked or something else didn't like, etcetera. Right? And a characteristic under this is that only a small subset of conflicts are detectable on inspection. Mostly, they are emergent, which means that they only appear when you try to come up with solutions that satisfy, all of them. Right? So, basically, it it's in the it's in the collective decision action of a bunch of different agents that you see that there is conflict. Right? So you can't rely on things like, oh, in my workflow, I did intent inspection, and I resolved all my conflicts, because that's not where you're gonna catch most of them. There are we've identified 10 families of coordination mechanisms that are that are out there in the in the application and research field. And to some degree, they depend on they depend on two things, really, their applicability. Are the decision and action loci within essentially a common place where you can link them, evaluate them simultaneously, that and are they under some kind of coordinated or similar authority, or are they really independent and sovereign? Right? According to those two axes, you have different methods that can help you. For example, you may, in some cases, simply have scope for imperative arbitration or normative regulation. On a very decentralized kind of sovereign agent, thing which may, conflict on use of resources, you're going to need to rely on something like markets. Right? There is no centralized coordinating, aspect. In in the middle, and if you're putting all your quasi you know, nominally independent network management agents within the same structure, then you can use this family of structured reconciliation, kind of, methods. Whether that's a, you you know, joint utility kind of calculation and optimization, some kind of negotiation, some kind of you know, there's a big family of distributed constraint optimization processes that that you can use. But, you know, it gets to this point that the the architecture of distribution of agents says a lot about what kind of conflicts and how you may be able to resolve them. I should say that cross cutting, there's this notion of if, you know, if you can use something like a digital twin to look at, okay. I'm gonna do everything all these guys wanna do. What have I got? Right? That's kind of a general fail safe. That can get you at your that's not only a technique you can use within within the structured reconciliation cluster, but it's also the only way you're going to get at things like, order and impact, conflict detection. So you wanna conceive of a structure that's got, you know, multiple pieces. A digital twin is probably going to play a big role generally as an exploratory leg before commit and a diagnostic post, commit. You know, you want to look at hopefully, lot of your agents are within the same network management structure. You can coordinate decisions and use this use techniques that are suitable to that. The NMOP draft is the one we were pointing to, and of course, it it it allowed for many permutations of where the agents could be placed, which is what tweaked this in the first place. The the kind of advice is not not adopt the NMOP inside MD, but focus on that first because that's kind of the the natural mode. There's you know, it's much easier to deal with conflict detection resolution preemption in that in that sphere. Again, there are some that, you know, you're going to have agents outside the the actual what they call a controller, what I, as more of a CSM guy, might call a management domain, but but there you go. So this should be thought through because it has implications on architecture. And, potentially, if we're defining interface models, are there communications related to coordination that need to figure in? This may be, you know, something that can be deferred, but I think eventually should be should be brought in. Okay. That's it for that one.
[01:10:13] Jean-Francois Tremblay: Thank you. I think Alex is in the queue. I don't know if it was for the previous presentation or this one. Okay. Alex, we we cannot hear you if you
[01:10:32] Alex Clemm: Oh, sorry. Can you hear me now?
[01:10:34] Jean-Francois Tremblay: Yep. Perfect.
[01:10:35] Alex Clemm: Okay. Sorry. Yeah. Yeah. So this was actually for the for the previous for the previous presentation. If I can ask this now. So I think this is actually so so I think this is actually a super interesting set of presentations. On on the earlier one, I would have the questions regarding the the the reconciling the models and so forth. It seems that a lot of this is concerned with when you want to read or observe or monitor information. So basically, perhaps assurance set of operation. But in since young models were mentioned specifically, there are also constraints that are sometimes expressed. Right? You have you have you have constraints. You have conditions. You have, yeah, dependencies and so forth. Do your techniques are you also considering those? Have you looked at them? Do they matter? Or are you looking, basically, first at some functions before you tackle those types of problems?
[01:11:34] Christophe: Probably for now, I'll just say it's it's it's very early days. We picked a a case to try, which is kind of an obvious one, a bit of a workhorse, just to to see what a workable process flow might look like and where the you know, whether there's prospect of success. You know, can the can the agents do enough thinking and learning to sort of bridge gaps? Is there a basis to continue? There there there seems to be, but it's it's very early days. So I'd love to talk to you offline or anyone else who's interested in this problem. Obviously, there there there was there were various flavors of this manifested in different contributions and different side meetings. This week, there was there was a side meeting on, you know, Yang modeling in the AI age or for AI systems, which touched on some of that same general grounds. So I think there's there's definitely the scope to do something different in an AI gentic world. So but it's very early days. So, you know, for specific questions, let's let's maybe talk offline, if that's okay.
[01:12:41] Alex Clemm: Cool. Thank you.
[01:12:44] Christophe: Okay. This last one, I'll I'll do fairly quickly. I don't want to overstate my welcome, and the key points, can be made, very easily. Okay. So this one is is saying, we talk a lot about, you know, performance on demand and dynamic networking requirements. And, you know, we're trying to create an automated smart operation system that, you know, can do can, you know, can take intent, can take a change in intent, can, you know, do service assurance, and can cope with these kind of dynamics. And we sort of neglect the problem of, well, we assume that a service intent is coming into the system, but how is it coming into the system and from whom and and why? Right? In a fully automated dynamic environment, you can conceive of agents on the demand generation side, not just the DSS, but the end, service user. You know, something that's saying, look, I'm I'm an AI environment or this other cloud computing environment, and I have dynamic distributed workloads. And I I wanna you know, I manage that, and I wanna be able to pilot my network services dynamically directly. Now you've got an agent to agent conversation flow that bridges not only the consumer provider, boundaries, but does that potentially in a commercial context, right, where you've got a service provider and a service purchaser. And this isn't completely addressed. What we wanted to show here is that we actually don't need any more models to do this. There's only one sort of new piece in the agent, ecosystem that's necessary to automate this. And actually, you all know what it is. Just the basic situation, you've got some enterprise or other customer system which is piloting its dynamic networking requirements. It knows why it needs them, etcetera. On the other side, you've got, a network system which is able to provide those, in a in a smart way. It's able to dynamically plan for services. It's able to assure them once once committed. It's able to yield information that may be relevant to pricing like, you know, resource scarcity information. The BSS is what connects them. You want that to be intent negotiation across the board. And, you know, here in IETF, we tend to talk about intent strictly as a data model. In other places, it's also talked about as a process flow, right, where there is a negotiation to get to the agreed intent. So we invoke some of that from TM Forum and Etsy. The the missing piece is, you know, the BSS is really central here because it's the thing that basically gates, the completion of a negotiation and and the actuation that sort of concludes the contract. But for that, there has to be sort of established agreement to the terms on the part of the customer. And the way that's done generally is with the digital wallet. Right? Something that says, hey. You know, for this purpose for these purposes, these are my priorities. These are my spending envelopes, etcetera. If you're within those rules, which I can update, but those rules are your guidance, right, when you're negotiating. So, I mean, I'll skip over the the, use case example and demo that's in the that's in the contribution and and skip right to the end. The the bottom line is that, you know, again, at ITF, we think a lot about models, but sometimes process is the completion to the solution rather than a new model. So in fact, we have all the models we need from different organizations to make this happen, including the the, agentic or digital, wallet piece. So we're definitely using the, service and then in the OSS, the other, network, etcetera, models that had been developed in IETF. But for the negotiation processes, and some of these other surroundings, we're using, we're using elements from from elsewhere. And what we found is there's nothing missing. So it may be a useful lesson as as there's more attention in some of the ITF groups on, hey. We wanna do this kind of dynamic I mean, I I heard this discussion this week. If we wanna support these kind of dynamic services, do we need a new Yang model that allows us to define the the parameters of the dynamics? Well, our contention is no. It's don't solve it with a a model. You you you have a model which can define the service you need at any point. The point is you've got to agree on what you're executing. And if you change your mind or have new demands, you're presenting that as a separate request. So it's it's a process overlay, not a requirement for a new model. Okay. That's it. Love to have any feedback on that.
[01:18:17] Jean-Francois Tremblay: Thank you, Chris. Feedback or question? Okay. I I have one Okay. Which which is more general comment and feedback because you present a lot
[01:18:31] Christophe: of
[01:18:31] Jean-Francois Tremblay: interesting stuff and a lot of questions and problems that needs to be solved. I was wondering actually two questions. The first one is what your expectation from the NMRG on this question. Do you want to ask for some support for some help in some of these questions? And the second one is more about the question, Gerard, that you would like to address. For example, if you think about, as you mentioned, would see conflict addressing conflict in a joint TKI with different actions. It's something that is not specific to our domain. And I know you also look into the what are domain, into the AI community as well?
[01:19:19] Christophe: Yeah. And I don't think there's there's one answer. I mean, as I said on the first topic, it's clear that there's some forming sense that there is an interesting macro topic there. You know, there there's discussion this morning about, will we have a new AI ops group? So what kind of agenda? Some colleagues ran a a side meeting on Wednesday that touched on these aspects. So it's forming. But, clearly, there are some there's agreement. There are some possibilities. They're worth exploring. It's not clear where in the organization the focal point might be. But, you know, I thought I'd introduce it here because, we are a research group for the moment. It's it's a research presentation. But certainly, you know, if you're interested, like, get I'll I'll make sure that the conversation around this is on the AI ops mailing list. It's probably a good focal point, you can jump in, and we can work. Probably, the next step is how do we get to a a topic either within a working group or something that's focused on this area, and then we can get sharp about what we're trying to solve. On the the the second one, I think it's more, when certain things are happening around agents, there's now a group of people that are cognizant of some things that need to be paid attention to and potentially leveraged. And again, if you're looking for interesting research problems, it's a very good area from what I'm seeing. The third one is more like, I think it's it's just a lesson that, you know, young models may be our hammer, but that doesn't make everything a nail. Right? It's a new model is not necessarily the solution to every problem. This was a case where, you know, a desirable utility scenario was fulfilled by working on process over top of existing models. But in but key models that are not only from iETF, but that are composed from different parts of the ecosystem.
[01:21:25] Jean-Francois Tremblay: So Okay. Yes. I think, Laurent, you have a quick question. Quick.
[01:21:35] Stefano Secci: Quick answer.
[01:21:37] Laurent Ciavaglia: Do you want a joke also or not? Because it's not a question about quick. It's question about presentation. Okay. It's more about the second presentation you had. You were describing a number of coordination patterns, and I think some of them are already kind of solved or addressed. What, from your point of view, will be the remains open for research in terms of coordination mechanism or, I don't discovery? What what would be a bit like where to look into?
[01:22:05] Christophe: Yeah. So, I mean, we we we got connected with people who are, for example, discovering the limits of the coordinated approaches like DCOP and and utility and looking at the mechanics of market based methods even within domains that you you wouldn't think you you need to use them or they're not they're not natural. There's a lot of these guys are doing some really hard math to sort of sketch the boundaries of utility and methods, etcetera. So there there is def and then and then even within the known methods, I think their application is not necessarily straightforward. So I think there's a lot to become expert on, a lot of research going on, people that can apply that to our systems, I think you know, the the reason we brought it up here or thought it was worth doing is just that, you know, people are conceiving of nominally independent agents in architectures, and there are real implications they should be aware of. And we at least need to think through how that might bound the architectures we we recommend. And if we're we're we're developing augmentations to protocols for their communication, is there something related to coordination that we need to anticipate? The answer may be no. It may be just another generic field you can you can undefined field you can carry, but but maybe. Anyways, we we thought it was was interesting just to to bring forward.
[01:23:39] Laurent Ciavaglia: Thanks, Hod.
[01:23:41] Jean-Francois Tremblay: Thank you. So next speaker. Okay. I think it yeah. Thales, you are the next one. No. Alex will present. Okay.
[01:23:57] Alex Clemm: Hello? Can you hear me?
[01:23:59] Jean-Francois Tremblay: Yes. Perfect. You just put your Okay.
[01:24:10] Alex Clemm: All right. So hello, everybody. This basically, this is a fairly new draft that I'm presenting, energetic AI for intent based networking. This is the outcome actually of a series of discussions that Toerless and I have had over, yeah, over actually, for a while and we thought it would be worthwhile actually to to to capture this and in in this draft and present this here. Let me see. How do I advance the slide? I guess here. Alright. So, basically, this follows up on RFC ninety three fifteen. Basically, this is where the intent based networking concepts and definitions were were defined, where, of course, intent being the set of operational goals the network should meet and outcomes that the network is supposed to deliver, what we define in a declarative manner without specifying how to achieve or to implement them. And, of course, this sounds a lot of what also Pagentic AI and so forth and large language models and so forth seem to be enabling. And so given the recent explosion of these, the question is, basically, does it make sense to revisit some of those questions or basically see how can we how can we leverage this to move closer to to, yeah, to to intent based network? Which and particularly, also, for instance, which functions in an intent based network stand to benefit? And also, are there any updates and so forth needed? And a similar question actually pertain also to autonomic networking. So basically, here, autonomic networking, as defined in ANIMA and IBM, kind of like work hand in hand. And there is actually a companion draft also shared by by Toerless in the ANIMA working group. And one way in which they go together is always, well, intent is it is recognized as a component, a necessary component of autonomic networks to provide guidance. So just as a reminder, so basically, this is basically the draft is is cycled basically structured around the life cycle and reference architecture that we have in ninety three fifteen, which basically lists a number of functions of both how they tie tie in with each other. And now, basically, in discussing the the role of agentic AI with this, we can basically look at each of those and the interaction and see, basically, how they might apply. And so, basically, looking at some of them, actually, the the the first ones, particularly the translation refinement step, that seems actually a very natural fit for agentic AI. I think Christopher's presentations just prior to this actually are are very indicative and actually very relevant for this for this space as well. And okay. And so likewise, basically, the whole aspect of orchestrating intent is and seems to be a natural fit for IdentityAI planning capabilities. There are various functions that can benefit from those that we are outlining basically in the draft. Basically, as a general rule, I guess, the leverage Algenic AI translation capabilities make it also easy to mask interface heterogeneity, which simplifies, of course, some of the biggest headaches in network language. It always has to do with dealing with differences, variations models and so forth, actually, again, tying in very nicely to the presentation that Christopher just gave. However, there are a number of gaps that are clearly also recognized, for instance, when it comes to not just translating and rendering steps, but we're actually developing smarter strategies to optimize networks as well as dealing with rainy day scenarios where, basically, we we need to for intent based networking to to be effective, you really need to have the network to stay in character and allow the operator to interact with it just in terms of intent, not by reverting to, for instance, to need to know or deal with very low level device parameter settings, for instance. And then finally, of course, there's a whole assurance part of the loop where, again, abstraction and reporting are genetic AI strengths that can be leveraged in various ways. There are also some severe practical still. And I think one thing to point out there specifically concerns the data velocity. Simply basically, you want to do this in real time, the amount of data that comes to events, to syslogs, to telemetry, through other data streams, and so forth, can be expected to overwhelm the ability of LLM infrastructure, program processing capability for the near future without any yeah, ways to deal with this. So basically, with this, basically, the questions are here also why we're presenting here in NMRG is basically, first, the question is, is this a discussion that is worthwhile having in NMRG? Basically, you should and the question is also how can NMRG contribute shaping this space? Basically, the way we see it, this is actually a very fast moving area, and a lot of the stuff in agentic AI is happening kind of like with or without IETF. And the question is basically and products will be also can be expected to emerge. But the question is what role can we still play into and help us to shape the discussion. Likewise, basically, how do we well, should we pursue this draft further and refine it further? There is a lot of other work also that is here, but kind of like it's all a little bit unstructured, and the connection is missing. And, of course, they all look at slightly different aspects of things. And, anyways, this is a starting conversation starter, if you will. We're looking for collaborators. Please contact us. Thurles is also in the room. I'm I'm remote, but both of us on email. And that's that concludes what I have to say.
[01:30:35] Jean-Francois Tremblay: Thank you, Alex. We do not have so much time now for question because we're training a lot of time. But of course, I will encourage people to, yeah, to go on mailing list or contact you. Just a a quick comment from from my side and and and and personally, but also on chat. So we, of course, identified identified that with this AI era, IBM clearly is a is a topic that is a rush to consider in the in the start of research in in network management. And that's why we I think it was in Montreal, we organized a kind of session around this topic somehow again to revisit this with with ISO. Yes. Probably, we continue to investigate or we can also continue on this in an MRG, of course, depending on the support we have. Yeah. Thank you. Thank you for the presentation.
[01:31:29] Toerless Eckert: Can I just make sorry, Jason? Just a quick comment. Right? So collaboration, for example, if you have examples of pieces of that solution where you already are using AI, right, so those things would be very nice to capture in the document as well just to understand the current state, not generically, but also by a practical example.
[01:31:50] Peter Boers: Sure.
[01:31:54] Jean-Francois Tremblay: Okay, thank you. So next presentation. Keep this. This is one's cook. Okay. Yes. The next presentation is from Shang Yu.
[01:32:20] Shang Yu: Yes. Okay. Thank you. Good morning, everyone. I'm Shang Yu from Shanghai University and the laboratory. Today, I will introduce our new draft, which is draft-xu-nmrg-idp-framework.
[01:32:32] Chengsu: Not speaking on the microphone, please.
[01:32:34] Jean-Francois Tremblay: Oh,
[01:32:34] Shang Yu: okay. And our goal is to make the intelligent processing in the data plan and safe and manageable. Okay. So the network data plan is changing quickly. And in the past, the data plan is mainly forwarded packets. But today, the programmable devices can collect information from the traffic and analyze traffic in at line rate. This creates new opportunities for AI based net network management. Our pre war previous research shown on the right is has demonstrated that such processing is possible. However, these systems use different interfaces and data formats. They also face similar problems with resources, state, and security. IDP builds on this research and provides a common framework for addressing these problems. So what do we mean by IDP? IDP processing has three important properties. First, it might be bounded. In other words, it cannot use unlimited time and resources. It must also use follow network policy. A decision function can only perform unauthorized action. Finally, it must be observable. The operator should be able to see what is happening. The process itself is simple. The node casts the signals and extracts features. If then runs as decision function and produces a result. If this process fails, the node enters safe mode and continues basic forwarding. The basic process gives us the architecture assurance here. The processing path moves from the signals to features and then to resolve to to to the decision and finally to action. I we identify three interfaces. The northbound connect the IDP nodes based the network management systems, and the East West interfaces connect the different I p IDP nodes. And finally, the internal interfaces will connect the forwarding engine to accelerator. These are logical interfaces, and we are not proposing three new protocols. Across these interfaces, IAP use three types of informations to communications with these systems. The first one is signal, and the second one is features, and the last one is results. The signals describe what is happening in the network. They may come from the packets, the flows, and the devices. Nodes use these signals to calculate features and a decision function that use these features to produce a result. The result may show a traffic class anomaly or a confidence pro confidence value. Two rules are important here. A result is normally only a hint. Network policy decides whether it can affect forwarding. Also, all the information should not be used for a new forwarding decision. The decision function does not have to use a specific AI algorithm. We describe its processing in three steps. The first one is partition, which divides the input into smaller parts. And the second one is map, process each part. And the finally, summary deals combines these results. Because this model model is general, it can be implemented on different hardware. A switch may use tables, while an FPGA or SmartNIC may perform a more complex processing. In every case, the decision function must have a clear identity and a version. The node must also know how many resources it needs. These functions also need to communicate with other parts of the system. So the node's prompt interface, the node reports its capability, and the manager system can then install the support policy. So the east device interface, IDP nodes can share information or transfer a task. The internal interface. The forwarding engine sends a worker to our accelerator and receives the result. Existing protocols can support this interaction, and our communication must remain within the policy and the trust scope of the IDP domain. Not all processing needs to happen in the same time. Simple and time sensitive decisions will be deployed on the fast path, and more difficult cases can be sent to an accelerator or external system. We call this escalation escalation. For example, if the local result has low confidence, the node may request a more detailed analysis. However, escalation must be controlled in in maximum rate and time out. Another protocol issue is memory. The IDPA functions need to remember information about a flow. However, forwarding devices have limited memory. The node, therefore, needs clear rules for managing this state. Any team may also use use it mechanisms to carry metadata. This draft doesn't define a new metadata format. Future work must explain how the information is protected and what happens when a node does not understand it. These design choices are summarized in 24 requirements. For clarity, we organize them into five areas. The first two areas is improved and decision function. And the second the the next two cover communication and the state's main state management. The final area covers the security and privacy. Across all five areas, the main principle is the same. An IDP function must stay within its results and policy limits. Security is especially important because an IDP result may change how traffic is handled. For example, an attacker may provide false signals or install a harmful model. An attacker may also try to use too many resources. Node must therefore check the source and the integrity of information information. A decision function must network policy. IDP must also protect privacy even without reading the packet content. Traffic patterns may reveal sensitive information. IDP is not intent to replace existing network mechanisms, and existing work supports network management, memory, traffic steering, and data export. At this stage, the document is just a framework. It doesn't define a new packet header or a new protocol. Specific protocol methods can be developed by developed later. So that's all. Thank you. Any questions?
[01:39:58] Jean-Francois Tremblay: Thank you. Unfortunately, we cannot take questions now for for lack of time. But, yeah, I just encourage people to to ask questions on mailing list or in the chat. But thank you for your for your presentation.
[01:40:09] Shang Yu: Okay. Thank you.
[01:40:15] Jean-Francois Tremblay: Thanks. The next one is Shailesh. I'm sorry. We we have to reduce the visit time for the No problem. Implementation.
[01:40:36] Prabhu: Yeah. Hi. This is Prabhu from Nokia. I'm going to introduce a framework on draft-prabhu-nmrg-prompt-schema-llm assisted network management. So over the past year, we have seen tremendous efforts in, you know, integrating LLMs into network management operations. And even in the context of ITF and the ITF, we have seen many drafts that have been, you know, working on, you know, NMA, a two a, and LLMs integrating into the network management systems. Right? So one of the key challenges that we're gonna face is the multivendered network elements are going to represent represent the same data in different languages. Right? And when this data reaches the the LLMs as prompts, because of the nuances in the representation, you know, there there's a risk of misinterpretation by the LLM, which is which can lead to the mismanagement of the network. Right? So just to quote an example, say, when the a can say show interface, when the b can say, you know, display interfaces. Right? And when does he can say interface brief? So all of them are the same intent, but, you know, they're different syntax, terminology, field names, and hierarchy. Right? So here's the current workflow. So there's vendor specific data. It could be alarms, telemetry, you know, configuration mess you know, messages, operational response, anything. Right? So because they're vendor specific, there are vendor specific ways of of prompt design and parsing. Right? That is fed into the LLM. Right? And because the prompts and parsing was logic was vendor specific, then we have AI applications that tend to be more on the vendor specific side as well. Right? So this limits a lot of things that including, you know, portability, interoperability, scalability, and, you know, prompt reuse. Right? So here is the proposed frameworks. Right? So what I propose is an input normalization layer that sits in between the network multivendor network elements and the central element that is managing the network. Right? So it starts with an input classifier, which is essentially a small language model, right, which helps us classify whether the data is performance data or configuration data or an operational response. Right? And based on that, it is fed into the prompt schema generator. So the schema is not restricted to a specific format. It could be JSON. It could be XML or Yang structures. Right? So and finally, the the the schema the standardized schema is then fed into the LLM for further processing. Right? So this way, we receive consistent inputs to the LLM as prompts, which, you know, which helps us in common abstraction and, you know, better interpretation and better management of networks as a result. So, you know, feedback requested. Right? So I would like the working group the research group to let me know if this is in the interest of I mean I mean, is there interest in pursuing this problem within NMRG? And I'd like to request the chairs also to, you know, kind of consider this problem within the research group. Right? And I'm also looking for volunteers to review the draft and let me know if any, you know, deployment scenario should be considered or, you know, if there's gap that we need to fill in
[01:44:16] Chengsu: that draft. Right? Yeah. Questions?
[01:44:20] Jean-Francois Tremblay: Thank you. And you're already effective, so you have time for question. We still have three minutes for questions or feedback.
[01:44:35] Laurent Ciavaglia: Hey. So I think it's interesting work. And, David, I mean, there are some discussion, especially in this meeting about agent ops, AI ops Right. Monitoring. I think this could be an interesting contribution because it's kind of normalization function that can sit between different agent that needs to communicate on agent and a networking management system or even order order order functionality. So I think this could be an interesting work to push into different discussion area. I'm not sure where it stands into the network management research, but management or definitely.
[01:45:14] Stefano Secci: Yeah. Thank you.
[01:45:18] Jean-Francois Tremblay: Roberta?
[01:45:20] Roberta: Hi, folks. Roberta here. Just to say that I really like this idea and this approach of having something that might be neutral to, you know, to companies and might, you know, fit things properly inside of the schemas. And I'm really looking forward to be able to contribute and and to review this this work as well. Count on me. I'll I'll talk with you later.
[01:45:47] Chengsu: Yeah. Thank you so much.
[01:45:51] Christophe: Just very quick, Chris, Huawei. What you're talking about here is not at all orthogonal to the, first topic I presented. I mean, it's part and parcel of the general theme. And as I said, there's many manifestations or particular use cases or instances of that theme. There there have been quite a few here. There was a a dedicated side meeting swirling around it. So I'll make sure, as I said, I think that we'll figure out somewhere Yeah. Through the AIOps AIOps mailing list in that process, somewhere where work on that team should should be done. Yeah. Exactly, you know, what points to so make sure you're make sure you're on that mailing list.
[01:46:39] Chengsu: Sure. Yeah. Yeah.
[01:46:44] Diego Lopez: Apart from what Chris has said that I believe that there is a very close relationship on what he was presenting about the different ontology fields and these because at the end, we are trying to to reconcile ontology or or what? At least semantic fields. I'm not Yeah. Point is that I think there is apart from from identifying ways for standardization, there is a there is an interesting matter here, I would say, in terms of research that is precisely evaluating which are the most suitable techniques to be applied. The architecture, I mean, the architecture you have depicted is generally enough to fit many different approaches. But I think that coming here in the future with somebody or sending the draft with an analysis of, well, if you go for mechanisms for semantic mapping or you go for groceries, using groceries to cross terms, knowledge graphs, whatever is something that would be helpful in having an idea of what which which are the the the the tools in the box that we have for this.
[01:47:57] Chengsu: Okay. Yeah. Okay.
[01:48:01] Jean-Francois Tremblay: Thank you. Thank you very much. So next speaker, please. The next speaker is Chengsu.
[01:48:21] Chengsu: Hello, everyone. So how many minutes to help? Many minutes to help?
[01:48:29] Jean-Francois Tremblay: Seven minutes.
[01:48:30] Chengsu: Seven minutes. Okay. Hope you take. Hello, everyone. I'm here on behalf of our call center team, and I will print our draft update on draft-wmz-nmrg-agent-ndt-arch. Actually, this is our fourth dashboard version, and we have presented the previously on at ITF one two three, four, and one five meetings respectively. And we have also discussed the talk topic in MRG interim meeting, March of this year. And I show six. Although thanks to our contributors and reviewers for comments and the strategies from perspective of operator operator as another technical points. And as soon as you know, we have built GitHub to tackle issue and PRs, and everyone is you're welcome and to report continuous PRs to report. And a quick status update and what's new for this version. And, actually, we did a lot of changes in the in dashboard. And here, I want to point highlight four points four major points first to that. We shifted from previous multi agent system to hybrid agent systems. This is to support the collaborative integration of agent skills, like CPIs and tools, as well as natural natural digital twin. Secondly, we streamlined the act architecture focus by what's removing the natural elements, the agent, and tag from the core architecture and move it to appendix for future exploratory. It's made our architecture is more clear and more focused on system level of currently software level or controller level. And third is that we strengthen the background and finding the instruction section with com comprehensive background context, including natural geotrain, adjunct AI, and the the ability that AI can bring to natural operations. Third fourth is we depend the gap and the challenges, including young adapt adoption and to agile AI, agent benchmarking, and the ability of agent observability control and intervention. So looking to the architecture itself, and there are four core characteristics in the architecture. The first is hybrid hybrid multi agent system. This is to feature a single NoSpend agent in at the top and below it is the multi multi task agents and as well as the planning tools and some some other interfaces as well. You you as as required, whatever you want to add it here. Second is that natural dual team, it can be a deterministic safety sandbox to validate agent decision in a risk free environment before configure changes or policy is deployed to the real network. As as you know, that natural digital team is also developing this emerging research group, and it has reached a broad consensus and now it's it's in provision process. And we think that with the joint or or convergence with giant AI, natural digital team can also play even bigger role in offer network operation and management. Third is that we enhanced our possibility and the human loop in the architecture. This is to tackle the AI black box dilemma and to to reduce some uncertainty issue or or some emergency issue with necessary human inter intervention. The fourth is knowledge graph. Knowledge graph is the semantic integration. This is to high to unify high heterogeneous data sources to build complex multi dimension correlation in across all the entire network. And when we do that, this is this can help agent to make better decisions and to reduce some holognese holognese holognese holognese holognese Okay. Based on the architecture, just on last Saturday and Sunday, we run a Hexagon demo on identity architecture and observability. You can see that just part of our whole whole picture, a whole architecture. In in this Hexum, we are to try we to validate the core components including agent system and observability functionality in our architecture. And we have to the the root cause analysis scenario, and and we have we can get to the chase its metrics and logs always edge edge in the execution, and we use large language model chi q one and some environment set to and use LM as charged to give the comprehensive evaluation result. And this is this this slide shows us the our hexagon results. You can see the accuracy is good and the talking cost. The time consumption is also affordable and will have very good result. And going forward, we can extend Hexagon with some more tools and and models involved, like natural and some knowledge graph and some other components used in our architecture. So what's next? First, we are calling the community feedback, and we are we are we we hope that more experts and viewers can provide more feedback. And we are here to request for research group adoption for this draft. Some reasons, first, we're seeing that this draft is better aligned with current emerging charter, And this chart is fill is filling cut pitot gap of high level architecture showing how a journey I facing through natural management. This is a strategic direction hole. It's definitely from Office ID. And just hours ago, we have assigned meeting. We also assigned meeting and a lot of topics is related to this topic. So we think that is actually our job can lay a foundation for some future discussion or researches. That's more that's all for my presentation. Thank you.
[01:56:09] Jean-Francois Tremblay: Thank you very much. Sorry, Chris. We we do not have time because we are still one last presentation. So, yes, the first point is very important before moving forward that you people may please provide feedback on mailing list so that we can we can provide on that.
[01:56:24] Chengsu: Okay. Thank you. Thank you. Chris, maybe you can send the question in the chat. Thank you.
[01:56:37] Jean-Francois Tremblay: Yes. So last presenter, Ming Seng. Okay.
[01:56:48] Sheng: How everyone. I'm from. The topic I'd like to share is about draft-nmop-cui-nkg-gateway. The knowledge graph is a active topic in ML, and we can see a lot of discussions on the collaboration between ML and MRD. So I think the these drops are provided useful bridge between the MRG. Okay. Let's start. So so the network knowledge graph ecosystem is becoming maturing. And there are lots of discussions around the architecture, the data unification, like, from YAM to KG and the main applications, like the type of modeling and the incident management. So I think we are moving from the basic concepts to operational domains. But there is still a critical gap. The another graph construction pipeline is ready, but the interaction is still hard. And most specifically, how how can users and applications interact with another graph but without any another graph operation as per test? So we figure out two points. The first is that the operator needs deep knowledge of the underlying graph schema and entity type and the queries and has select c four or SPARQL, which are a specific neural graph of querying language. The second thing is that there is a mismatch between high level usage type annotation and the low level graph operations. So the core changer we're focusing in this draft is to translate the high level request to low level operations. And and we give us three three key requirements. The first is that we need to handle the ingenious inputs. And the second is we need to normalize the high level Internet to intermediate representation. The final one is we need to ensure security. We propose a three layer architecture. In the upstream layer, it's post process input and the interactions. In the midstream layer, it translates and resolves the hello intent to a level operations. The last layer, the downstream layer is responsible for storing the fundamental data in our graph. So in the upstream layer, it's supposed to cancel for imports. The first is the user input, which is expressed by the natural language. The second one is application input, which may be represented as a domestic a specific language. In the midstream layer, it it receives a hello intent and and they're translated into the internal catalog and and it's heading intents. And in the downstream layer, it stores the network graph network graph and spot some of our graph operations. Here, we gave a use case. In this in this case, our operator may ask a special HTTP traffic attack affecting our specific service. So as it as operator inputs our natural language query, I know we can trust this query into our as a c four. And this c four can be executed on the non graph and since that is the final results and returning it as a neutral as a summary or has, you know, your neutral long draw or some structural data and returning it to to the user. Oh, okay. That's how I mess. That's okay.
[02:00:41] Jean-Francois Tremblay: Yeah. Thank you for your your effective presentation. Thank you very much. So it's the the end of an emergency session to send that. So please, yes, also yes to react on mailing list for the last presentation that we did not have so much time for question. And again, you for participating to this session and see you soon.
[02:01:08] Chengsu: In San Francisco.
[02:01:11] Jean-Francois Tremblay: Or maybe I'm tearing before. I am. If you're going this regards Nissan's very new Cisco. Please go. Yeah. No.
[02:01:24] Alex Clemm: That's enough.