Markov Decision Problems in ATM Traffic Control

Abstract: This thesis discusses how to make cost-effective use of the communication resources in the Broadband Integrated Services Digital Network (B-ISDN), which is based on the Asynchronous Transfer Mode (ATM) switching and multiplexing technique.The thesis deals with two important functions in ATM traffic control, namely Call Admission Control (CAC) and routing, which affects both the network operator's revenue over time and the users' Quality of Service (QOS) and Grade of Service (GOS). The routing function finds a route, expressed in terms of successive links, with sufficient QOS (e.g. cell loss probability) according to the CACQOS function. The CACGOS function accepts or rejects the call request based on fairness (e.g. call blocking probability) and revenue considerations.The CACGOS and routing tasks are modelled a Semi-Markov Decision Problem (SMDP). The SMDP solution gives high resource utilization and ability to control GOS distribution between the call classes. In SMDP routing, the task is to control the state transitions between reward generating states such that the average reward rate is maximized. In order to obtain a solution with feasible computational complexity, the network SMDP is decomposed into a set of link SMDPs. Each link SMDP is solved by either dynamic programming (DP) or reinforcement learning (RL). DP is based on a model of the decision task in terms of the state transition probabilities and expected reward in each state. RL is not based on a model of the decision task. Instead, the optimal policy is found from simulated state transitions, where long-term reward predictions are corrected by temporal difference learning.We study aspects such as delayed set up of wide-band calls, link-level integration of guaranteed QOS services and best effort services, and Poisson versus self-similar call arrival processes.