## Abstract

With network functions decoupled from specific hardware, network function virtualization (NFV) is a promising technology to accelerate service provisioning in datacenter (DC) networks. In inter-datacenter elastic optical networks, each virtualized network function (VNF) is usually deployed in multiple DCs for the sake of survivability. In service provisioning, different VNF selections greatly affect IT resources in DCs as well as spectrum resources in optical networks. This paper investigates how to select appropriate VNFs for service requests to achieve joint load balancing of IT and spectrum resources. Two joint balancing factors are proposed to quantify the impact of different VNF selections on network load. Furthermore, a Joint-Optimization Selection (JOS) algorithm is designed to select VNFs in a joint load balancing manner. The performance of the proposed algorithm is evaluated in terms of resource utilization, blocking probability and average path length. Simulation results indicate that the proposed algorithm outperforms the benchmark algorithm.

© 2019 Optical Society of America under the terms of the OSA Open Access Publishing Agreement

## 1. Introduction

With network function virtualization (NFV) technologies, network functions can be decoupled from dedicated hardware devices and realized in the form of virtualized network functions (VNFs) [1]. Thus, Capital Expenditure (CAPEX) and Operational Expenditure (OPEX) can be significantly reduced for network operators, and the deployment of new services is also accelerated. The inter-Datacenter (inter-DC) network has become an important infrastructure to provide data storage and information processing resources [2]. Operators can provide a variety of services and predefine VNFs in particular order according to service-level agreement (SLA). For example, video stream usually needs to pass through transcoder and compressor, which can be realized via VNF [3]. To meet the high capacity transmission, such as data backup and virtual machine (VM) migration [4,5], elastic optical network (EON) is employed for the interconnection among different DCs. Compared with wavelength division multiplexing (WDM) technology [6–8], EON achieves agile bandwidth allocation and significantly improves the spectrum efficiency by dividing a standard ITU-T grid into fine-grained spectrum slots [9,10].

In inter-DC EONs, each VNF instance is usually deployed in multiple DCs from the perspective of network survivability. For each service, the traffic is steered from source to destination and processed by a set of VNFs. Thus, each service connection consumes IT resource in DCs as well as spectrum resource along the lightpath from source to destination. For each VNF, operators have to choose appropriate DCs hosting the desired VNFs from the candidate DCs. Different selections of VNF correspond to different usage of IT and spectrum resources, which further affects the network load [11]. To be specific, if VNF selection only considers IT resource in DCs, the usage of spectrum resource in fiber links is not optimized. In the worst case, even though DCs have enough IT resources, network congestion will make some DCs unreachable. Similarly, if VNF selection only focuses on the optimization of spectrum resource in EONs, some DCs might easily get overloaded and the desired VNFs become unavailable. Therefore, the problem addressed by this paper is how to select VNFs to achieve the joint optimization of IT and spectrum resources in inter-DC EONs. It is necessary to note that most of existing articles focus on the optimization of VNF placement and routing and spectrum allocation (RSA). Different from the network scenarios in the existing articles, this paper assumes that the VNFs are already deployed in DCs. The objective is to select the appropriate VNFs from multiple candidate DCs for each service to realize the joint optimization of IT and spectrum resources.

The concept of Global Balancing factor (GB-factor) and Local Balancing factor (LB-factor) were proposed in our previous work [12] to evaluate the load status for each DC. LB-factor focused on the adjacent links of DC where VNF was deployed, whereas GB-factor considered the k-shortest paths through DC. Based on these two factors, a Joint-Optimization Selection (JOS) algorithm was designed to solve the problem of VNF selection. This paper extends our previous work and main contributions can be summarized as follows.

- • The simulation is optimized in this paper. The benchmark is changed from random selection algorithm to non-joint optimization selection. Comparing the joint optimization with non-joint case, simulation results can directly show the benefits of joint optimization. Besides, a new metrics, i.e., average path length, is introduced in the simulation.
- • The proposed algorithm has been optimized for the scenario where each service connection needs to be processed by multiple VNFs. The previous scheme only considers the case of one VNF. Then the simulation has been updated accordingly.
- • Different network scenarios are considered to evaluate the performance of the proposed algorithms in this paper. Previous work just focused on the impact of different traffic loads. On the basis of previous work, this paper also evaluates the performance with different ratios between spectrum and CU usage.

The rest of this paper is organized as follows. Section II provides a brief survey about the related works. Section III introduces network model and formulates the problem of VNF selection in inter-DC EONs. Then the joint balancing factor and JOS algorithm are proposed in Section IV. Section V evaluates the performance of JOS algorithm in two network scenarios. Finally, Section VI summarizes this paper.

## 2. Related work

With the outstanding advantages introduced in the white paper [13], NFV has aroused great interests in academia and industry field. Most studies on VNF or service function chain (SFC) deployment can be classified into two cases, i.e., offline case and online case. The offline case means the static network planning problem, which is usually formulated as an integer linear programming (ILP) or mixed ILP model [14]. All the requests are assumed to be known in advance. In the online case, the requests arrive and depart dynamically and the target is to achieve efficient provisioning of VNF according to service requests [18].

The work in [14–17] have addressed the problem of VNF placement and provisioning in different network scenarios. For multicast services in inter-DC EONs, the work in [14] investigated how to orchestrate NFV trees, and designed a mixed integer linear programming (MILP) model and a heuristic algorithm based on path-intersection (PI). In packet/optical DC, Xia *et al.* [15] addressed the problem of VNF deployment to minimize the cost due to optical/electrical/optical (O/E/O) conversions. In a multi-domain EONs, Wang *et al.* [16] investigated how to cost-effectively provision vNF graphs (vNFGs) with arbitrary topologies, with the objective to minimize the total resource cost. In DC networks, a 2-phase algorithm for chaining VNFs was proposed in [17] to achieve joint network and server load balancing.

The VNF or SFC deployment can be realized with different constraints and optimization objectives. The study in [18] jointly optimized the deployment of new users’ SFCs and the readjustment of in-service users’ SFCs to achieve the balance between resource consumption and operational overhead. A game-theoretic approach with dynamic resource pricing strategy was proposed in [19] to optimize the provisioning of SFCs and improve the network-wide profits. A column generation model was introduced in [20] to solve the problem of minimizing network energy consumption while satisfying the SFC requirements. Considering the dependence constraint of VNFs, the strategy of SFC embedding was investigated in [21] to make efficient use of the substrate resources. To reduce the blocking probability, Sun *et al.* [22] studied the problem of service chain deployment and designed an online forecast-assisted algorithm to predict future VNF requirements. Similarly, the SFC provisioning procedures in [23] were divided into pre-deployment phase and provisioning phase. In the pre-deployment phase, a deep-learning (DL) model was designed to predict future SFC requests.

It is notable that the SFC consumes IT resources in DCs as well as spectrum resources in EONs. Different allocation policies of IT and spectrum resources affect the efficiency of resource utilization in substrate networks. The work in [24] designed an ILP model and a heuristic algorithm to realize efficient VNF service chaining through jointly allocating spectrum and IT resources in inter-DC EONs. However, this work is fundamentally different from ours due to the following two aspects. On one hand, they only addressed the static network planning problem while we focus on the dynamic network scenario. The service requests dynamically arrive and depart in this paper. If the requests cannot be served immediately at arrival time, they will be blocked. On the other hand, the network model in [24] is different from ours. Specifically, they considered joint optimization of VNF deployment and spectrum allocation while we investigate how to select appropriate VNF instances in terms of load balancing after VNFs are deployed in DCs.

## 3. Problem formulation

This section mainly defines the network scenario and formulates the problem in this paper.

#### 3.1 Definition of network scenario

In this paper, we focus on the dynamic traffic in inter-DC EONs, which means the service requests dynamically arrive and depart. Each DC connects to an optical node in EONs. Note that the link connecting DC with optical node is assumed to have infinite capacity. Except the electric regeneration in DCs, the transmission is transparent in EONs. Each type of VNF is instantiated and deployed in multiple DCs in terms of survivability. For each request, one connection from source to destination should be established with the required bandwidth. Meanwhile, each request needs to be processed by at least one VNF. The number of VNFs assigned to the service depends on the service requirements. For simplification, this paper assumes that each service request needs one or two VNFs and different type of VNFs have the same demand of IT resources. Thus, resource allocation for service request includes IT resource in DCs and spectrum resource in EONs. This paper assumes that the VNF processing does not change the bandwidth requirement of service. The aim of this work is to minimize the blocking probability of services by optimizing the VNF selection for services. The notations and corresponding definitions in this paper are listed in Table 1.

#### 3.2 Problem statement

Figure 1 illustrates the problem of VNF selection in the inter-DC EONs. The objective of our work is to optimize the VNF selection for new coming services in order to minimize the blocking probability. We assume that there are 6 frequency slots (FSs) in each optical link and 100 computing units (CUs) in each DC. The service request is denoted as $r(A,D,\{{f}_{1},{f}_{2}\},2,40)$. According to the definition, the source and destination are node A and D, with the required VNF ${f}_{1}$and ${f}_{2}$. The spectrum and IT resource demand of service is 2 FSs and 40 CUs, respectively.

This example presents two different VNF selection schemes for VNF ${f}_{1}$. As shown in Fig. 1, VNF ${f}_{1}$ is deployed in DC_b and DC_e. Thus, two options are available to select the VNF ${f}_{1}$. In option I, the DC_b is selected and the path from A to D is A$\to $B$\to $C$\to $D. The load status of optical links and DC are shown in Fig. 1(a). We can find that part links and nodes (e.g. link A-B, B-C and DC_b) are in high load status whereas the other links and nodes (e.g. link F-E, E-C and DC_e) are almost idle. By contrast, as shown in Fig. 1(b), option II selects DC_e to provide the required VNF. The path from A to D is A$\to $F$\to $E$\to $C$\to $D. Although the path length increases by one hop, this option makes the network load more balanced for both IT resources and spectrum resources. It is notable that for two options in Fig. 1, the spectrum allocation from source to destination does not need to meet the continuity constraint. With the O/E/O conversion in DC, an end-to-end path is divided into several lightpaths, where spectrum allocation can be performed independently. For instance, resource allocation indices for path A-B-C-D change over each link in Fig. 1(a) while the resource allocation for path A-F-E is different from path E-C-D in Fig. 1(b). It is important to note that contiguity constraint must be satisfied in the spectrum assignment for each path/sub-path. In Fig. 1(a), the spectrum blocks colored with red are allocated for this request. We can see that the allocated spectrum blocks are contiguous from node A to node D. The same goes for the connection from node A to node D in Fig. 1(b).

In this case, we can see that option II is more effective to select ${f}_{1}$ in terms of load balancing. This illustration explains that the VNF selection greatly affects the load status in inter-DC EONs. Therefore, it is meaningful to consider joint balancing of IT and spectrum resources during the VNF selection.

## 4. Joint balancing factor and Joint-Optimization Selection (JOS) algorithm

To accomplish VNF selection and service provisioning in inter-DC EONs, there are two problems to be solved. The first problem is to select appropriate DCs to provide the desired VNFs. The second is to establish paths from source to destination by allocating IT and spectrum resources. We propose the concept of joint balancing factor to quantify the impact of different VNF selections on network load. Then a heuristic joint-optimization selection algorithm is designed to optimize the VNF provisioning.

#### 4.1 Local balancing factor and global balancing factor

Each type of VNF could be deployed in multiple candidate DCs. Considering both IT resources in DCs and spectrum resources in optical links, joint balancing factor $\phi (n)$ is proposed for the VNF selection. According to different locations of spectrum resources, $\phi (n)$can be classified into two types, global balancing factor (GB-factor) and local balancing factor (LB-factor). For each candidate DC, $\phi (n)$ reflects the impact of different VNF selections on the load status in inter-DC EONs. The smaller value means that if the VNF in this DC is selected, new coming service has less influence on the load status of IT resources and spectrum resources.

In Eq. (1), the joint balancing factor $\phi (n)$consists of two parts ${\phi}_{cu}(n)$and ${\phi}_{fs}(n)$. The former is the ratio of required IT resources to all the available resources in DC_n. The calculation of ${\phi}_{cu}(n)$is the same for GB-factor and LB-factor whereas the calculation of ${\phi}_{fs}(n)$is different. We use ${\phi}_{fs}^{GB}(n)$ and ${\phi}_{fs}^{LB}(n)$to denote the ${\phi}_{fs}(n)$ in the GB-factor and LB-factor, respectively. ${\phi}_{fs}^{GB}(n)$considers the k-shortest paths through DC whereas ${\phi}_{fs}^{LB}(n)$ just focuses on the adjacent links of DC.

In LB-factors,${\phi}_{fs}^{LB}(n)$ is calculated according to Eq. (2). In this case, we consider the available spectrum resources on all the adjacent links of DC. The value of coefficient$\alpha $ depends on the location of DC node. If the DC node is source or destination node, α is set to 1. Otherwise, the coefficient is 2 since the service consumes the spectrum resource in both ingress link and egress link.

In GB-factor, ${\phi}_{fs}^{GB}(n)$ calculation is related to the location of DC node. If the DC node is source or destination, Eq. (3) and Eq. (5) are used to calculate${\phi}_{fs}^{GB}(n)$. Otherwise, we should separately consider the path from source to DC node and from DC node to destination based on Eq. (4) and Eq. (5). Note that the first part in Eq. (5) is the average length of the k-shortest paths from ${v}_{1}$to ${v}_{2}$.The second part represents the ratio of required slots number to average number of free slots on each link. Thus, the product of two parts can reflect the load status of the k-shortest paths between two nodes.

Figure 2 shows the calculation of GB-factor and LB-factor. The available frequency slots (FSs) of each link are marked in the seven-node topology. For the service request$r(A,G,\{{f}_{1}\},2,20)$, the source and destination are node A and G, respectively. The demand of spectrum and IT resources is 2 FSs and 20 CUs. The required VNF ${f}_{1}$ has been deployed in DC_b and DC_d, which are connected to node B and D.

We first calculate the GB-factor and LB-factor for ${f}_{\text{1}}$ deployed in DC_b. The value of ${\phi}_{cu}(b)$is the same for both factors. With 50 CUs available in DC_b, ${\phi}_{cu}(b)$is equal to 20/50. Equations (1) and (2) are used for the LB-factor ${\phi}^{LB}(b)$. Since DC node is an intermediate node, $\alpha $ is set to 2. Meanwhile, the available spectrum resources on all the adjacent links of DC_b are 3, 4, 3, 2 FSs. According to the calculation shown in Fig. 2, ${\phi}^{LB}(b)$is 0.73. Equations (1), (4) and (5) will be used for the GB-factor ${\phi}^{GB}(b)$. With $k=2$, two candidate paths from A to B are A-B and A-F-B. For path A-B, the path length is one hop and three slots are available. Similarly, for path A-F-B, the length is two hops and there are three common free slots (i.e. #3, #4 and #6) on these two links. For the path from B to G, two candidate paths are B-G and B-F-G. The former has one hop and three free slots whereas the latter has two hops and four free slots. At last, the value of ${\phi}^{GB}(b)$is 2.22. Similarly, ${\phi}^{LB}(d)$ and ${\phi}^{GB}(d)$can be calculated with the above method. Assume that the available IT resources in DC_d are 100 CUs. For the calculation of ${\phi}^{LB}(d)$, there are five and six available FSs on its adjacent links. The path A-B-C-D and A-F-E-D are two candidate paths from A to D while D-C-G and D-E-G are two options for the path from D to G. Finally, we can get the results of ${\phi}^{LB}(d)$ and ${\phi}^{GB}(d)$.

The calculation of GB-factor and LB-factor have been elaborated from the example. Based on GB-factor, DC_b should be selected for VNF processing since ${\phi}^{GB}(b)<{\phi}^{GB}(d)$. However, DC_d is preferable for the service in terms of LB-factor with ${\phi}^{LB}(d)<{\phi}^{LB}(b)$. Hence, different balancing factors directly affect the VNF selection, which leads to different network performance.

The calculation of GB-factor in Fig. 2 is for the request with one VNF. Two calculation methods are available for the case of multiple VNFs. One option is separate calculation for each VNF while the other option is to calculate the GB-factor by jointly considering multiple VNFs. In terms of reducing calculation complexity, this paper adopts the method of calculating GB-factor separately for each VNF.

#### 4.2 Joint-Optimization Selection (JOS) algorithm

Based on the above factors, a novel Joint-Optimization Selection (JOS) algorithm is designed for the VNF provisioning in inter-DC elastic optical networks. The pseudo-code of the algorithm is shown in Table 2. Without loss of generality, we take a service requiring two VNFs as an example and the VNF set is ${S}_{vnf}=\{{f}_{1},{f}_{2}\}$. For services requiring more than two VNFs, the VNF provisioning can be realized similarly.

JOS algorithm consists of two main parts, i.e., VNF selection and routing and spectrum allocation (RSA) for service requests. In the first part, we check all the candidate DCs to provide the required VNFs. If the candidate DC has enough CU resources to meet the service requirement, the joint balancing factor $\phi (n)$is calculated. Note that the factor in the algorithm refers to GB-factor or LB-factor. For the request with multiple VNFs, the GB-factor is calculated separately for each VNF to reduce the calculation complexity. Then, for each type of VNF, all the available DCs are sorted in ascending order of $\phi (n)$. Different DC sequences result from different joint balancing factors in the algorithm.

In the second part, routing and spectrum allocation is carried out for each service. Since the connection from source to destination must go through the selected DCs, we modify the Dijkstra algorithm to calculate the constrained shortest path. Note that considering the complexity in the calculation of GB-factor, the shortest path is considered in this part to reduce the complexity of JOS algorithm. The first fit policy is used for the spectrum allocation. Note that with the O/E/O conversion in DCs, resource allocation for each sub-path can be performed independently. For each required VNF, all the DCs in ${S}_{DC}^{i}$are traversed. If spectrum resources can be found on all the sub-paths, the connection will be established successfully from source to destination. Otherwise, the service request is blocked.

#### 4.3 Time complexity analysis

We focus on the time complexity of LB-factor and GB-factor calculation in the proposed algorithm. The time complexity of LB-factor calculation is $O(fs\times 2{N}_{E}/{N}_{V})$,where $fs$ is the slot number of each fiber link. ${N}_{E}$ and ${N}_{V}$ refer to the number of edges and nodes in the network. Thus, $2{N}_{E}/{N}_{V}$stands for the average node degree in the network. For GB-factor, the time complexity of calculation is higher than LB-factor. The calculation of average length of the k-shortest path is $O\left(K\times \left|V\right|\times \left(\left|E\right|+\left|V\right|\mathrm{log}\left(\left|V\right|-1\right)\right)\right)$. We assume that the number of links is approximately $\mathrm{log}\left(\left|V\right|-1\right)$for each path. The time complexity of calculating the common free slots on the k-shortest path is$O\left(K\times fs\times \mathrm{log}\left(\left|V\right|-1\right)\right)$. Therefore, the total time complexity of calculating GB-factor is$O\left({K}^{2}\times \left|V\right|\times \left(\left|E\right|+\left|V\right|\mathrm{log}\left(\left|V\right|-1\right)\right)\times fs\times \mathrm{log}\left(\left|V\right|-1\right)\right)$.

## 5. Simulation and results analysis

As shown in Fig. 3, USNET is used as the simulation topology, consisting of 24 nodes and 43 links. The parameter configuration is listed in Table 3. We consider three types of VNFs in the simulation and each VNF is deployed in three different DCs. All the DCs are attached to optical nodes with high node degree (i.e., node degree is 4 or 5). In terms of reducing network cost, only several nodes are assigned as DCs. In addition, the schemes of VNF placement vary with different optimization objectives (e.g. energy efficiency, network cost). Thus, there is no universal solution for VNF placement. The placement policy used in this paper is just one option among them. Each DC can accommodate 3000 computing units (CUs) and each fiber link has 320 frequency slots. Simulation results are obtained by averaging 20 experiments, where 50000 service requests are generated following Poisson distribution and their holding time follows exponential distribution. All the presented results have a confidence interval not exceeding 6%, with 95% confidence level. The source-destination pair of each service is randomly chosen from all the nodes. With QPSK modulation format, the number of required frequency slots is distributed between one and eight. For each service request, the Dijkstra algorithm is extended to compute the constrained shortest path. The first fit policy is used for spectrum assignment. We assume that each service needs to be processed by one or two VNFs, which are randomly chosen from three types. For each service, the number of required CUs for VNF processing is proportional to the number of spectrum slots.

The performance of the proposed algorithm is evaluated through simulation experiments. We focus on four performance metrics in this paper, i.e., connection blocking probability, CU utilization ratio, bandwidth utilization ratio and average path length. This paper focuses on dynamic traffic, where service requests dynamically arrive and depart. If the request cannot be served immediately at the arrive time, it will be blocked. The blocking probability is the ratio of blocked request number to the total request number. The average path length refers to the average value of path length for all the served request. Note that blocking probability and average path length are calculated after all the requests arrive. Bandwidth utilization refers to the ratio of occupied bandwidth to the total bandwidth in the network. Similarly, CU utilization is represented by the ratio of occupied CU to the total CU. Since both CU and bandwidth utilization ratio vary with time during the simulation, we record the CU and bandwidth utilization every 5000 services. Then, with 50000 requests in total, CU and bandwidth utilization are the average value of 10 samples in each simulation. Note that CU and bandwidth utilization are recorded every 15000 services in our previous paper [12]. With a shorter record interval, more samples are available to calculate the resource utilization, leading to the improvement of statistical correctness. Two network scenarios are considered in the simulation. The first case is to investigate the impact of different traffic loads on the performance of JOS algorithm. The second case is to take into account different ratios between CU and bandwidth usage for each request. In addition, the benchmark algorithm is to select VNFs only based on available IT resources in DCs.

#### 5.1 Performance evaluation in different traffic loads

In this case, traffic load increases from 500 to 700 Erlangs with the increment of 25 Erlangs. Note that traffic load is calculated as the ratio of the mean holding time to the mean inter-arrival time of connections. For each service request, the ratio between CU and bandwidth usage is set to 5:1.

Figure 4(a) shows the connection blocking probability of three schemes. We can see that with traffic load from 500 to 700 Erlangs, both GB-based JOS and LB-based JOS can achieve lower blocking probability compared with benchmark. Meanwhile, GB-based JOS shows more benefits than LB-based JOS. For instance, if the traffic load is set to 575 Erlangs, nearly 9% of service requests are blocked with the benchmark algorithm, while for GB-based and LB-based JOS, the blocking ratio can be reduced to 3% and 7%, respectively. It is interesting to note that GB-based JOS and LB-based JOS can achieve performance improvement in different traffic loads. The reason is that both factors consider joint load balancing of IT resource in DCs and spectrum resource in optical networks. The difference is that GB-factor considers the available spectrum resources on three candidate connections whereas LB-factor only focuses on the local resources in adjacent links of DC.

Figure 4(b) shows the results of CU utilization ratio in different traffic loads. It is clear that with the increase of traffic load, more CUs are needed to meet the requirements of service requests. We can also observe that both GB-based and LB-based JOS consume more IT resources than the benchmark algorithm. In addition, for the same traffic load, GB-based JOS occupies more IT resources than LB-based JOS. For instance, if the traffic load is 575 Erlangs, 62% CUs are needed with the benchmark algorithm. By contrast, the GB-based JOS and LB-based JOS occupy about 67.8% and 64.3% CUs respectively. The reason is that GB-based JOS achieves lower blocking probability than LB-based JOS. The larger number of connections deployed in networks leads to the higher IT resource utilization.

The results of average path length in different traffic loads are shown in Fig. 5(a). We can see that with the shortest path length in average, the results of GB-based JOS are about one hop shorter than benchmark scheme. In addition, the results of LB-based JOS are slightly higher than benchmark scheme. This can be explained as follows. By considering joint balancing, GB-based JOS has better VNF selection than benchmark scheme. It is also interesting to notice that the average path length decreases with the increase of traffic load for all the three schemes. The reason is that with the increase of traffic load, the connection requests with longer length has the higher probability to be blocked due to the lack of resources.

The results of bandwidth utilization ratio are shown in Fig. 5(b). We can see that the bandwidth utilization ratio in all the three schemes increases with the growth of traffic load. It is important to note that bandwidth usage is related to the product of three main factors, i.e., average slot number of each connection, average path length and the number of deployed connections. Since slot number for each connection is randomly distributed from 1 to 8, we can assume three schemes have the same value for the first factor. The product of average path length and the number of deployed connections determines the bandwidth usage. The results on the second factor is shown in Fig. 5(a). GB-based JOS has the shortest average path length whereas LB-based JOS has the longest one. The third factor, i.e. the number of deployed connections, is determined by the blocking probability in Fig. 4(a). Thus, GB-based JOS has the maximum value and the benchmark has the minimum value on the third factor. Consequently, GB-based JOS has the lowest bandwidth usage.

#### 5.2 Performance evaluation with different ratios between CU and bandwidth usage

Even though having the same bandwidth demand, different services have different requirements on the CUs, which is determined by service property. In this case, the traffic load is fixed at 600 Erlangs and the ratio between CU and bandwidth usage is set to different values from 1 to 9. For the ease of description, the ratio between CU and bandwidth usage is denoted as symbol $r$.

The results on blocking probability are shown in Fig. 6(a). When $r$ increases from 1 to 4, three lines are relatively flat. The blocking probability of benchmark algorithm remains at about 9%. Both GB-based and LB-based JOS reduce the blocking probability compared with benchmark. The former reduces by 5% blocking probability while the latter brings about 1% reduction. If $r$ is higher than 4, the blocking probabilities of three schemes increase with the growth of $r$. Especially, when the $r$ is set to 9, three algorithms almost have the same results. The reason is that when the value of $r$ is relatively low (e.g. less than 5), the ratio of required CU resources to total CU capacity is much smaller than the ratio of required bandwidth to total bandwidth capacity. In other words, bandwidth resources have much higher impact on VNF selection than CU resources. For each scheme, the blocking probability mainly depends on the available bandwidth resources. However, if $r$ is larger than 4, the CU resources become the decisive factor of blocking probability. The higher value of $r$ leads to higher CU usage for each service, which results in higher blocking probability.

The results on CU utilization are shown in Fig. 6(b). All the three schemes occupy more CU resources with the increase of $r$. When the $r$ is fixed, GB-based JOS has the highest CU utilization ratio while the benchmark algorithm has the lowest one. The lower blocking probability means the higher CU utilization. For instance, for $r$ = 6, the CU utilization of GB-based JOS is about 82% while the other two schemes need about 79% and 73% CU resources, respectively. We can also find that LB-based JOS achieves the similar performance with the benchmark on the CU utilization when $r$ is smaller than 5. This is because the blocking probabilities of these two schemes are close to each other and the maximum difference of two schemes is about 1%.

Figure 7(a) shows the results of average path length for three schemes. Considering the joint load balancing of CU resource and bandwidth resource, the average path length of GB-based JOS is the shortest. The results of LB-based JOS are close to the benchmark and the maximum difference is about 0.25 hop. In addition, for each scheme, the average path length remains stable if $r$ is lower than 6. In this case, the bandwidth resource is the major determinant of average path length. However, when the ratio $r$ increases from 6 to 9, the value of average path length will increase. The higher value of $r$ results in higher CU demand for each service. Consequently, less services can be served by local DCs and some requests are processed in the remote DCs. That is why the average path length increases when $r$ is from 6 to 9.

The results of bandwidth utilization are shown in Fig. 7(b). We can see that when $r$ is from 1 to 5, the bandwidth utilization of three schemes fluctuate slightly. To be specific, GB-based JOS occupies about 46% bandwidth resources, which is the least among three schemes. By contrast, LB-based JOS and benchmark scheme need about 50% and 48% bandwidth resources, respectively. As mentioned in the analysis of Fig. 5(b), bandwidth usage is related to the product of average slot number of each connection, average path length and the number of deployed connections. The first factor can be assumed to be the same for three schemes. For the second factor, as shown in Fig. 7(a), GB-based JOS has the lowest value. The third factor is determined by the blocking probability in Fig. 6(a) and GB-based JOS has the largest value on the number of deployed connections. Finally, for GB-based JOS, the product of three factors is minimum among three schemes. We can also find that if the ratio $r$ is higher than 5, the bandwidth utilization of three schemes decrease gradually. This can be explained as follows. As shown in Fig. 6(a), the blocking ratio increases sharply when $r$ is from 6 to 9, which means that less and less connections are deployed in the networks. Although the average path length increases from $r$ = 6 to $r$ = 9, the increment is limited to about 0.5 hop. Thus, the product of three factors decreases when $r$ increases from 6 to 9.

## 6. Conclusion

This paper investigates how to select appropriate VNFs for services to achieve joint load balancing of physical infrastructure resources in inter-DC EONs. GB-factor and LB-factor are proposed to quantify the impact of different VNF selections on network load. Based on these two factors, a novel JOS algorithm is designed to optimize VNF selections as well as determining the RSA for the services. The performance of the proposed algorithm is assessed in terms of blocking probability, resource utilization as well as average path length. Two network scenarios are considered in the simulation. Simulation results show that GB-based JOS outperforms the LB-based JOS and benchmark algorithm. By considering the joint load balancing of IT and spectrum resources, GB-based JOS achieves the reduced blocking probability and improved network performance.

## Funding

National Science and Technology Major Project (Grant No. 2017ZX03001016), and National Natural Science Foundation of China (NSFC) Project (Grant No. 61822105, 61571058 and 61601052).

## Acknowledgments

Part of this work has appeared in the proceeding of International Conference on Optical Communications and Networks (ICOCN), Wuzhen, China, 2017 [12].

## References

**1. **J. Garay, J. Matias, J. Unzilla, and E. Jacob, “Service description in the NFV revolution: trends, challenges and a way forward,” IEEE Commun. Mag. **54**(3), 68–74 (2016). [CrossRef]

**2. **L. Peng, M. Chen, K. Park, and C. H. Youn, “Virtual-Pod-Assisted routing and resource assignment in elastic all-optical intra-datacenter networks,” IEEE Access **5**, 406–420 (2017). [CrossRef]

**3. **N. T. Jahromi, S. Yangui, A. Larabi, D. Smith, M. A. Salahuddin, R. H. Glitho, R. Brunneri, and H. Elbiaze, “NFV and SDN-based cost-efficient and agile value-added video services provisioning in content delivery networks,” Proc. CCNC, 671–677 (2017). [CrossRef]

**4. **P. Lu, L. Zhang, X. Liu, J. Yao, and Z. Zhu, “Highly efficient data migration and backup for big data applications in elastic optical inter-data-center networks,” IEEE Netw. **29**(5), 36–42 (2015). [CrossRef]

**5. **W. Kabaciński, M. Michalski, R. Rajewski, and M. Żal, “Optical datacenter networks with elastic optical switches,” Proc. ICC, 1–6 (2017). [CrossRef]

**6. **A. Muhammad, N. Skorin-Kapov, and L. Wosinska, “Manycast and anycast routing for replica placement in datacenter networks,” Proc. ECOC, 1–3 (2015). [CrossRef]

**7. **B. R. Rofoee, G. Zervas, Y. Yan, and D. Simeonidou, “Griffin: programmable optical datacenter with SDN enabled function planning and virtualisation,” IEEE J. Lightw. Technol. **33**(24), 5164–5177 (2015). [CrossRef]

**8. **H. N. Tan, T. Inoue, K. Tanizawa, T. Kurosu, and S. Namiki, “All-optical Nyquist filtering for elastic OTDM signals and their spectral defragmentation for inter-datacenter networks,” Proc. ECOC, 1–3 (2014). [CrossRef]

**9. **M. Gerstel, M. Jinno, A. Lord, and S. J. Yoo, “Elastic optical networking: a new dawn for the optical layer?” IEEE Commun. Mag. **50**(2), s12–s20 (2012). [CrossRef]

**10. **Y. Zhao, B. Chen, J. Zhang, and X. Wang, “Energy efficiency with sliceable multi-flow transponders and elastic regenerators in survivable virtual optical networks,” IEEE Trans. Commun. **64**(6), 2539–2550 (2016). [CrossRef]

**11. **M. S. Yoon and A. E. Kamal, “NFV resource allocation using mixed queuing network model,” Proc. GLOBECOM, 1–6 (2016). [CrossRef]

**12. **B. Li, X. Yu, Y. Zhao, Y. Li, Q. Ou, Z. Liu, X. Liao, and J. Zhang, “Joint IT and spectrum resource load balancing for VNF selection in inter-datacenter elastic optical networks,” Proc. ICOCN, 1–3 (2017). [CrossRef]

**13. ** Network Functions Virtualization – Introductory White Paper, [Online]. Available at: https://portal.etsi.org/Portals/0/TBpages/NFV/Docs/NFV_White_Paper3.pdf

**14. **M. Zeng, W. Fang, J. J. P. C. Rodrigues, and Z. Zhu, “Orchestrating multicast-oriented NFV trees in inter-DC elastic optical networks,” Proc. ICC, 1–6 (2016). [CrossRef]

**15. **M. Xia, M. Shirazipour, Y. Zhang, H. Green, and A. Takacs, “Network function placement for NFV chaining in packet/optical datacenters,” IEEE J. Lightw. Technol. **33**(8), 1565–1570 (2015). [CrossRef]

**16. **Y. Wang, P. Lu, W. Lu, and Z. Zhu, “Cost-efficient virtual network function graph (vNFG) provisioning in multi-domain elastic optical networks,” IEEE J. Lightw. Technol. **35**(13), 2712–2723 (2017). [CrossRef]

**17. **M. T. Thai, Y. D. Lin, and Y. C. Lai, “A joint network and server load balancing algorithm for chaining virtualized network functions,” Proc. ICC, 1–6 (2016). [CrossRef]

**18. **J. Liu, W. Lu, F. Zhou, P. Lu, and Z. Zhu, “On dynamic service function chain deployment and readjustment,” IEEE eTrans. Netw. Serv. Manag. **14**(3), 543–553 (2017). [CrossRef]

**19. **X. Chen, Z. Zhu, J. Guo, S. Kang, R. Proietti, A. Castro, and S. J. B. Yoo, “Leveraging mixed-strategy gaming to realize incentive-driven VNF service chain provisioning in broker-based elastic optical inter-datacenter networks,” IEEE/OSA J. Opt. Commun. Netw. **10**(2), A232–A240 (2018). [CrossRef]

**20. **N. Huin, A. Tomassilli, F. Giroire, and B. Jaumard, “Energy-efficient service function chain provisioning,” IEEE/OSA J. Opt. Commun. Netw. **10**(3), 114–124 (2018). [CrossRef]

**21. **M. Jalalitabar, E. Guler, D. Zheng, G. Luo, L. Tian, and X. Cao, “Embedding dependence-aware service function chains,” IEEE/OSA J. Opt. Commun. Netw. **10**(8), C64–C74 (2018). [CrossRef]

**22. **Q. Sun, P. Lu, W. Lu, and Z. Zhu, “Forecast-assisted NFV service chain deployment based on affiliation-aware vNF placement,” Proc. GLOBECOM, 1–6 (2016). [CrossRef]

**23. **B. Li, W. Lu, S. Liu, and Z. Zhu, “Deep-learning-assisted network orchestration for on-demand and cost-effective VNF service chaining in inter-DC elastic optical networks,” IEEE/OSA J. Opt. Commun. Netw. **10**(10), D29–D41 (2018). [CrossRef]

**24. **W. Fang, M. Zeng, X. Liu, W. Lu, and Z. Zhu, “Joint spectrum and IT resource allocation for efficient VNF service chaining in inter-datacenter elastic optical networks,” IEEE Commun. Lett. **20**(8), 1539–1542 (2016). [CrossRef]