Solving for Systems:-
and measure, measure, measure and characterise.
Jim Kurose explains the Internet to five different levels of complexity.
Traffic Management
This is mainly about the Timescale decomposition of traffic management components, packet transit/RTT times, flow setup times, traffic matrix time variations, and long term demand variation (usually increase).
Also novel protocol deployment create new demand patterns.
This all creates requirements for empirical input e.g. of user utilities for elastic and inelastic traffic demands, and for behaviour in response to offpeak (or congestion) type time varying charging for resource use. A great example of recent work on how things change sometimes quite quckly is this paper about the change in application demands during the pandemic
Signaling protocol complexity - signaling couples multiple components - end systems and users, to reuters/switches, schedulers and admission control, and even routing, so signaling is a mess, and to date, very little deployed in the Internet. Most traffic management decisions are made using bespoke (ad hoc) measurement tools and techniques.
One of the big arguments recently is how to divide up a budget (whether congestion or delay) amongst hops (e.g. AS hops) in a path - with some overall resource constraint (e2e delay must be less than 300ms) each hop will want to maximise the delay it can impose for bursty traffic (same for RED based ECN triggers) so it can maximise user traffic - need incentive matching!
Systems Design
Some things change demand surprisingly quick. - already mentioned the impact of the pandemic on increased in demand for interactive video/audio, but also sources shifted from work to working from home in daytime. New change is decentralised social media like Mastodon and Matrix, which have many p2p servers (mastdon today has around 4000, serving 8M people - the user base is growing at around 1M a week!)
Interesting social/legal/regulatory constraints include the simple (the EU mandates free data roaming or even trivial, like USBC phone charging sockets!) to the subtle - the Digital Markets Act requires any large service provider to open up APIs for interoperation - new work in internet standards means there will be open protocols that allow all systems (not just e-mail and web servers, but messaging and social media etc) to interwork - presumably also video conferencing
Technically, this is actually trivial (e.g. most video and audio use same coding - indeed in WebRTC have to use same protocols too). The trickier pieces are interworking key management for security, especially for group communications.
Pipelining example lead to replacement of HTTP 1.0/TCP with HTTP 3/QUIC, which allows arbitrary ordering of packets delivered over UDP (but still with reliability, flow and congestion control, and e2e privacy) - this allows browsers far more freedom to render material from multiple sources/media.
And this will keep changing and changing as people create new applications and services, and new communications technology - all that requires measurement, measurement and also measurement.
Data Centers - example of Queue Jump
1. Data centers have regular topology and software can be centrally managed
2. the Fan-in factor of traffic, sometiems known as TCP in-cast, can and does cause spikey delays, which very badly reduces the performance of distributed compiutations, clock synchronisation and in memory disk caches, all of which degrades the throughput of data centers
3. if we can differentiate traffic, we can use different schedules to treat flows with low latency requirements, from those with high throughput but no particularly latency bound needs.
4. in a 3 hop data center, a small number of priority queues will work - as long as the sum of traffic from sources of a given priority is controlled based on computing the occupancy of that priority class, and its delay impact - then there is capacity left for lower priority, and still very short queues too... Sources can be policed to ensure the classes don't starve out lower priorities, and this can be done below the app in the OS, or below the OS in a hypervisor
Optimisation:- multipath routes, decentralised rates.
from an entirely different perspective, we can treat the routing and rate control problems as optimisation challenges - in this approach, we can use gradient descent methods for assigning flows to routes, and distributed optimisation (via feedback control and increase/decrease searching for optimal utility) to compute rates.
Note well - the optimisation of routes can work at any level of aggregation, and hence is suitable for traffic engineering, and is largely a centralised technique. The formulation is also agnostic about multipath routing, so is suitable in the presence of live use of redundant paths and load balancers, and is consistent with end-to-end multipath protocols (e.g. multipath TCP or QUIC). One tends to think of the route optimisation approaches being for longer term matching of traffic to paths (but could also suit open flow controlled traffic at the individual flow level, obviously, though this is not used in the Internet today). Of course, gradient descent methods are widely used in training in machine learning and AI.
n.b. finding minimum of a function reminder
The distributed, asynchronous, non-coordinated optimisation of rates (i.e. TCP congestion control or equivalents) is also applicable to other distributed machine learning. The rate adaptation is also suitable for multipath end-to-end protocols (i.e. MPTCP). So the rate optimisation techniques operate on the round-trip time timescales.
I am going to loop back around to flow and congestion control, because these things go together like strawberries and cream, or meta and verse:-)
Flow control can be open loop (call setup with a traffic descriptor and an associated admission control algorithm), or closed loop, based on feedback (dupack, packet loss/timeout, or explicit congestion notification).
Packet forwarding can be FIFO, or Round Robin, or weighted according to some request (management, setup, payment, etc - out-of-band). If it is round robin, we get fairness quite naturally, and some degree of protection against misbehaviour of other flows. If the flows are policed (because they gave a descriptor in open loop, or because they implement congestion avoidance and control), then we get more protection against misbehaviour of other flows of packets. If the router implements an Active Queue Management scheme (like RED), as well as a schedule (like round robin), then we get more protection (even against our own misbehaviour).
A neat example of use of scheduling in a different layer is in the new web protocol, QUIC, in browsers - this paper illustrates nicely how one can improve rendering of pages by changing the order that components flow through the layers of HTTP, QUIC, to/from UDP/IP...
We've looked at unicast (one direction) multicast (some direction), broadcast (all directions), mentioned any cast (any direction), and mobile (one level of redirection. We've also looked at metric v. random based route choice, and some policy interactions with traffic engineering (preferences that differ from policy or metric).
Note - slide 228 lists R (trunk reservation) with opposite sense to r on y axes on graphs showing diminishing return of impact of trunk reservation
Next, we look at open-loop and closed-loop flow control. History of feedback controllers is ancient!
A fun thing to look at is the BSD TCP kernel code. - this book by Rich Stevens is a walk through of that!
