I. Introduction

Deep fiber access solutions have been deployed worldwide to push optical fiber closer to individual homes and businesses and to help realize different flavors of fiber-to-the-x (FTTx) networks, where x denotes the discontinuity point between optical fiber and some other wired or wireless transmission medium. Today’s broadband access networks leverage both optical fiber and wireless technologies with seamless convergence, giving rise to FiWi access networks [1]. FiWi access networks combine the reliability, robustness, and high capacity of optical fiber networks and the flexibility, ubiquity, and cost savings of wireless networks. Fully converged networks, where different fixed and mobile access technologies can be flexibly selected while sharing core network functionalities, will be instrumental in realizing future 5G low-latency applications. This is particularly advantageous for those use cases that do not necessarily require mobility all of the time and thus can be carried out in fixed broadband network environments [2].

Historically, the huge bandwidth potential of optical fiber, which is far in excess of any other utilized transmission medium, has lured most research efforts into focusing on the primary goal of continuously increasing the capacity of optical networks rather than on, for example, lowering their end-to-end latency. This comes as no surprise, given that a single strand of fiber offers a total bandwidth of 25,000 GHz, which can be easily tapped into using wavelength division multiplexing (WDM). To put this potential into perspective, it is worthwhile to note that it is about 1000 times the entire usable radio frequency (RF) spectrum on planet Earth [3]. As an illustrative example, Fig. 1 shows the next-generation passive optical network (NG-PON) roadmap as envisioned back in 2009, where the primary design goal for (r)evolutionary NG-PON1&2 broadband access networks was the provisioning of ever increasing capacity over time [4]. However, this perspective slowly started to change in 2013,¹ when questions surfaced about whether to focus access research efforts on more than just continued capacity upgrades [5]. According to [6], one of the major factors limiting the performance of edge mobile networks is latency.

 

Fig. 1. Next-generation passive optical network (NG-PON) roadmap as of 2009, illustrating the primary goal of continued capacity upgrades in the past [4]. ©2009 IEEE.

Download Full Size | PDF

PON technologies are anticipated to accelerate 5G deployments [7]. One solution to reduce latency in PONs is to modify the architectural structure of the remote node by adding loop-back fibers to the passive splitter. In doing so, a local Fx fronthaul (Fx-FH) can be realized for direct inter-optical-network-unit (ONU) communications, where ONUs may interface with their collocated macro and small-cell base stations and thus form local clusters for coordinated multipoint (CoMP) transmission in Long Term Evolution-Advanced (LTE-A) networks. Note that Fx is used to denote various lower-layer split points along the 5G radio processing chain, as specified by the ITU-T [8]. The author of [7] also emphasized the importance of end-to-end coordination of both PON and wireless network resources via a common orchestrator that runs one or more cooperative dynamic bandwidth allocation (co-DBA) algorithms in support of emerging 5G low-latency applications. A preliminary study of a distributed medium access control (MAC) protocol and simple DBA algorithm run by ONUs to support low-latency communication among base stations across multiple PONs, whose neighboring remote nodes were interconnected via additional fiber links, was presented in [9]. This study exploited PON-based technologies for realizing mobile backhaul infrastructures. Similarly, in [10,11], AT&T reported on their strategy to leverage existing PON-based fiber-to-the-node (FTTN) residential access, right of way, and already-installed powering facilities to provide inexpensive small-cell backhaul. Conversely, mobile fronthaul networks, which interconnect centralized baseband units (BBUs) with remote radio heads (RRHs) located at cell sites, have been implemented by using digital fiber-optic interfaces such as the Common Public Radio Interface (CPRI) [12]. For instance, China Mobile’s cloud radio access network (C-RAN) is CPRI based and hence makes use of digital radio-over-fiber (RoF) techniques.

Clearly, one way to realize the common orchestrator is by centralizing all end-to-end coordination functions in the cloud, giving rise to the widely studied C-RAN; see, e.g., [13–15]. C-RAN is able to achieve significant cost savings by sharing centralized network resource management units among mobile users. We revisit C-RAN in Section III and elaborate on its pros and cons in light of the emerging concept of edge computing [16]. Edge computing is a new paradigm in which computing and storage resources—variously referred to as cloudlets, micro-data centers, or fog nodes—are placed at the Internet’s edge in proximity to wireless end devices in order to achieve low end-to-end latency, low jitter, and scalability [17].

In addition to economic considerations and ultra-reliable and low-latency communication (URLLC) requirements, another important aspect of the 5G vision is decentralization. 2G-3G-4G cellular networks were built under the design premise of having complete control on the infrastructure side. According to [18], 5G systems should abandon this design principle and evolve the cell-centric architecture into a device-centric one by exploiting intelligence at the device side (human or machine) within different layers of the protocol stack. Similarly, unlike the previous four generations of cellular networks, 5G networks were understood from the start to integrate cellular and WiFi technologies and standards [19]. Recall from above that some 5G use cases do not necessarily require mobility all of the time and thus can be carried out in fixed broadband network environments. Hence, future 5G networks need to be fully converged networks, where different fixed and mobile access technologies can be flexibly selected while sharing core network functionalities, leading to latency and reliability improvements [2]. Towards this end, conventional 10G PONs (XG-PONs) and coarse WDM (CWDM) technologies were shown to be the most cost-effective optical transport solutions for rolling out 5G fixed wireless access [20]. Furthermore, to achieve a convergence gain by integrating fixed and mobile infrastructures, an important cornerstone in today’s network operator strategy is the use of a common transport platform in a fiber-to-the-home (FTTH) and small-cell roll-out scenario, including wireless access via WiFi, in support of future 5G low-latency applications [21].

In this paper, we focus on the emerging Tactile Internet as one of the most interesting 5G low-latency applications. In addition to conventional audiovisual and data traffic, the Tactile Internet envisions the real-time transmission of haptic information (i.e., touch and actuation) for the remote control of physical and/or virtual objects through the Internet. We introduce FiWi-based small-cell network infrastructures that naturally lend themselves to the implementation of decentralized co-DBA algorithms and inherent mobile edge computing (MEC) at their optical-wireless interface in order to usher in future 5G URLLC services by means of WiFi offloading and fiber backhaul sharing. Note that in September 2016, the European Telecommunications Standards Institute (ETSI) removed “mobile” from the MEC acronym and renamed it multi-access edge computing (MEC) in order to broaden its applicability to HetNets, including WiFi and fixed access technologies (e.g., fiber) [22]. One of the goals of this paper is to show how FiWi enhanced LTE-A HetNets with artificial intelligence (AI)-based MEC capabilities may help realize the aforementioned attributes of the 5G vision by studying the Tactile Internet as an illustrative use case.

The remainder of the paper is structured as follows. Section II briefly reviews the salient features and requirements of the Tactile Internet. It also derives Tactile Internet traffic models from haptic traces by studying teleoperation as an example of an immersive Tactile Internet experience. In Section III, we revisit FiWi access networks in the context of conventional clouds and emerging cloudlets, thereby highlighting the limitations of centralized C-RAN in light of future 5G networks moving toward decentralized architectures. Importantly, we elaborate on the implications of wireless access via WiFi on the design of decentralized co-DBA algorithms in support of future 5G low-latency applications. Section IV introduces the concept of low-latency FiWi enhanced LTE-A HetNets using advanced MEC with embedded AI capabilities; it also considers fiber-lean strategies for realizing local Fx-FH solutions for direct inter-ONU communications. Section V presents analytical latency results verified by haptic-trace-driven simulations. In Section VI, we outline some future research ideas that will help tap into the full potential of the Tactile Internet, paying particular attention to decentralized blockchain technologies. Section VII concludes the paper.

II. Tactile Internet

The term Tactile Internet was first coined by Fettweis in 2014. In his seminal paper [23], the Tactile Internet was defined as a breakthrough enabling unprecedented mobile applications for tactile steering and control of real and virtual objects by requiring a round-trip latency of 1–10 ms. Later in 2014, ITU-T published a Technology Watch Report on the Tactile Internet, which emphasized that scaling up research in the area of wired and wireless access networks will be essential, ushering in new ideas and concepts to boost access networks’ redundancy and diversity to meet the stringent latency as well as carrier-grade reliability requirements of Tactile Internet applications [24].

To give it a more 5G-centric flavor, the Tactile Internet has been more recently also referred to as the 5G-enabled Tactile Internet [25,26]. Recall from above that unlike the previous four cellular generations, future 5G networks will lead to an increasing integration of cellular and WiFi technologies and standards [19]. Furthermore, the importance of the so-called backhaul bottleneck needs to be recognized as well, calling for an end-to-end design approach leveraging both wireless front-end and wired backhaul technologies. Or, as eloquently put by Andrews, the lead author of [19], “placing base stations all over the place is great for providing the mobile stations high-speed access, but does this not just pass the buck to the base stations, which must now somehow get this data to and from the wired core network?” [27].

This mandatory end-to-end design approach is fully reflected in the key principles of the reference architecture within the emerging IEEE P1918.1 standards working group (formed in March 2016), which aims to define a framework for the Tactile Internet [28]. Among others, the key principles envision to (i) develop a generic Tactile Internet reference architecture, (ii) support local area as well as wide area connectivity through wireless (e.g., cellular, WiFi) or hybrid wireless/wired networking, and (iii) leverage computing resources from cloud variants at the edge of the network. The working group defines the Tactile Internet as follows: “A network, or a network of networks, for remotely accessing, perceiving, manipulating, or controlling real and virtual objects or processes in perceived real-time.” Some of the key use cases considered in IEEE P1918.1 include teleoperation, haptic communications, immersive virtuality, and automotive control.

A. Human-in-the-Loop-Centric Design

Clearly, the Tactile Internet opens up a plethora of exciting research directions towards adding a new dimension to the human-to-machine interaction via the Internet. According to the aforementioned ITU-T Technology Watch Report, the Tactile Internet is supposed to be the next leap in the evolution of today’s Internet of Things (IoT), although there is a significant overlap among 5G, IoT, and the Tactile Internet, as illustrated in Fig. 2. Despite their differences, all three share an intersecting set of design goals:

 

Fig. 2. Three lenses of 5G, IoT, and the Tactile Internet: commonalities and differences [29]. © 2016 IEEE.

Download Full Size | PDF

In [29], we elaborated on the subtle differences between the Tactile Internet and the IoT and 5G vision, which may be best expressed in terms of underlying communications paradigms and enabling end devices. Importantly, the Tactile Internet involves the inherent human-in-the-loop (HITL) nature of human-to-machine interaction, as opposed to the emerging IoT without any human involvement in its underlying M2M communications. While M2M communications are useful for the automation of industrial and other machine-centric processes, the Tactile Internet will be centered around human-to-machine/robot (H2M/R) communications and will thus allow for a human-centric design approach towards creating novel immersive experiences and extending the capabilities of the human through the Internet, i.e., augmentation rather than automation of the human [30].

B. Haptic Traffic Characteristics

An interesting example of a Tactile Internet experience that allows for remote immersion is the HITL-centric use case of teleoperation based on haptic communications. As stated earlier, the Tactile Internet envisions the real-time transmission of haptic information for the remote control of physical and/or virtual objects through the Internet [31]. Figure 3 illustrates a typical teleoperation system based on bidirectional haptic communication between a human operator (HO) and a teleoperator robot (TOR) in a remote task environment. The HO interfaces with the communication network (to be described in greater detail in Section IV) via the so-called human system interface (HSI). The HSI device is used to display haptic interaction with the remote TOR to the HO. The controllers on both ends of the teleoperation system ensure the tracking performance and stability of the HSI and TOR. A perceptual deadband-based (i.e., zero output if changes in consecutive samples are minimal) data reduction may be deployed as a lossy compression mechanism by exploiting the fact that human end-users are not able to discriminate arbitrarily small differences in haptic stimuli. The human perception of haptics can be exploited to reduce the haptic packet rate. Specifically, the well-known Weber’s law determines the just noticeable difference (JND), i.e., the minimum change in the magnitude of a stimulus that can be detected by humans [32]. Weber’s law gives rise to the so-called deadband coding technique, whereby a haptic sample is transmitted only if its change with respect to the previously transmitted haptic sample exceeds a given deadband parameter $d\ge 0$ (given in percent) [33].

 

Fig. 3. Teleoperation system based on bidirectional haptic communications between a human operator (HO) and a teleoperator robot (TOR) in a remote task environment.

Download Full Size | PDF

Note that despite growing interest in the Tactile Internet, there is still limited understanding of the characteristics of haptic traffic, especially at the packet level. For simplicity and analytical tractability, Tactile Internet traffic has been assumed to be Pareto or Poisson distributed in recent studies, e.g., [34]. In the following, we take a closer look at the specific characteristics of Tactile Internet traffic by studying the use case of teleoperation. Specifically, we study two sets of haptic traces obtained from teleoperation experiments involving TORs with different degrees of freedom (DoF). Note that the number of independent coordinates required to completely specify and control/steer the position, orientation, and velocity of a TOR is defined by its DoF. Currently available teleoperation systems range from 1-DoF to $>20$-DoF TORs. For instance, a 6-DoF TOR allows for both translational motion (in 3D space) via force and rotational motion (pitch, yaw, and roll) via torque. The two considered teleoperation experiments involve TORs with 1 and 6 DoF. Furthermore, our haptic traces comprise measurements with different values of deadband parameter $d$.

1) Teleoperation Experiments:

2) Packet Interarrival Times:

Next, we investigate the packet interarrival times of both teleoperation traces. First, we focus on the position and orientation samples in the command path (from HO to TOR). Let us assume that position-orientation sample $i$ is transmitted as a packet in the command path at time instant $T_i^{(c)}$. Thus, the corresponding packet interarrival times $I_i^{(c)}=T_i^{(c)}-T_{i-1}^{(c)}$, $i=2,3,\dots$, represent realizations of the random variable $I^{(c)}$. Figure 4(a) depicts the histogram of the packet interarrival times $I^{(c)}$ in the command path obtained from the 6-DoF teleoperation traces, with the most frequent packet interarrival time expectedly being centered at 1 ms due to the default haptic sampling rate of 1 kHz.

 

Fig. 4. Experimental 6-DoF teleoperation packet interarrival times: (a) command path and (b) feedback path.

Download Full Size | PDF

The histogram of the packet interarrival times $I^{(f)}$ in the feedback path (from TOR to HO) is shown in Fig. 4(b). Interestingly, the feedback path differs from the command path in that it exhibits two peaks at approximately 0.75 ms and another one at 1.25 ms. Upon examining the force/torque traces stemming from the TOR side, we found that the two peaks exist because the force and torque sensors of the TOR operate at two slightly different sampling rates above and below 1 kHz.

In an effort to find a probability distribution function (PDF) that best fits the experimental packet interarrival times in Fig. 4(a), we considered a variety of well-known distributions. Our preliminary evaluations narrowed our choice down to three candidate PDFs, namely, exponential, generalized Pareto (GP), and gamma distributions, as shown in Table I. Our method of selecting the best fitting PDF comprised the following three steps. First, we used the maximum likelihood estimation (MLE) method to estimate the parameters of each PDF, as listed in Table I. Second, the estimates of the first step were verified by computing the complementary cumulative distribution function (CCDF) $F_{I^{(c)}}(\zeta )=P(I^{(c)}>\zeta )$. Third, to compare the goodness of fit among the three PDFs under consideration, we used the maximum difference $D^{*}$ between the fitted and experimental CCDFs, which is given by

TABLE I. Comparison of Best Fitting PDFs for 6-DoF Teleoperation Packet Interarrival Times

View Table | View all tables in this article

Figure 5(a) shows the CCDF of the three fitted PDFs and experimental 6-DoF teleoperation packet interarrival times in the command path. We observe from the figure that the gamma distribution matches the experimental data reasonably well, as opposed to the exponential and GP distributions. This observation is further verified by the fact that the gamma distribution achieves the smallest value of $D^{*}$, as highlighted in bold in Table I. Similar to the command path, we observe from Fig. 5(b) and Table I that for the CCDF in the feedback path, $F_{I^{(f)}}(\zeta )=P(I^{(f)}>\zeta )$, the gamma distribution again best fits the experimental data.

 

Fig. 5. CCDF of fitted PDFs and experimental 6-DoF teleoperation packet interarrival times: (a) command path and (b) feedback path.

Download Full Size | PDF

Next, we study the 1-DoF teleoperation experiment, which included deadband coding unlike its 6-DoF counterpart. For fair comparison of the two sets of haptic traces, we post-processed the original 6-DoF traces and applied deadband coding for a variety of different deadband parameter values in the command path ($d_c$) and feedback path ($d_f$). We again determined the best fitting PDFs for the packet interarrival times with the different deadband parameter values by following the same approach as described above. Figure 6 comprehensively summarizes our findings on the different best fitting packet interarrival time distributions for command and feedback paths with and without deadband coding in both of the teleoperation scenarios. We observe that, in general, command and feedback paths can be jointly modeled by the GP, gamma, or deterministic packet interarrival time distribution, depending on the given value of deadband parameters $d_c$ and $d_f$, as shown in Fig. 6.

 

Fig. 6. Summary of best fitting packet interarrival time distributions for command and feedback paths with and without deadband coding: (a) 6-DoF teleoperation and (b) 1-DoF teleoperation.

Download Full Size | PDF

Importantly, our haptic trace analysis indicates that the assumption made in recent studies that Tactile Internet traffic is Pareto distributed is not valid for the analyzed traffic. Furthermore, the assumption of Poisson traffic (e.g., [34]) with exponentially distributed packet interarrival times was found valid only for 6-DoF teleoperation in the feedback path with deadband parameter values of $d_f\ge 15\%$. We note that our trace analysis provides important yet preliminary insights into the statistics of Tactile Internet traffic. Clearly, a more systematic approach looking at additional haptic traces of different types of teleoperation experiments will be instrumental in accurately validating the packet interarrival time distributions reported above.

III. FiWi Access Networks: Revisited for Clouds and Cloudlets

DBA is one of the contributing factors to latency in FiWi access networks, and thus it is important to understand how the edge architecture with its associated MAC protocol(s) affects the overall DBA strategy. Decentralized DBA is preferred in order to eliminate the delays inherent with a centralized scheme. This is especially critical for the latency-driven applications of the Tactile Internet.

Recently, the authors of [37] provided a preliminary study of the shortcomings of centralized optical line terminal (OLT)-based DBA algorithms in FiWi access networks consisting of a conventional time-division multiplexing (TDM) PON with a passive power splitter at the remote node and base stations (BSs) connected to ONUs. Each BS was assumed to wirelessly exchange its queue information with other BSs in an attempt to build and maintain global knowledge among all BSs and thereby facilitate the use of a distributed DBA algorithm among all ONU-BSs. In doing so, the OLT was exempt from the upstream bandwidth allocation process and thus avoided the detrimental impact of PON propagation delays in traditional centralized DBA algorithms. Despite the reported queueing delay performance improvements over the well-known centralized DBA algorithm IPACT, the wireless exchange of periodic control messages among all BSs may not be scalable in the wireless front end. Furthermore, subscribers may access the wireless medium without the network assistance of BSs in a truly distributed manner, as explained in the two architectures described next.

A. FiWi: EPON and WLAN

Although a few FiWi architectural studies exist on the integration of PON with LTE or WiMAX wireless front-end networks, the vast majority of studies consider FiWi access networks consisting of a conventional IEEE 802.3ah Ethernet PON (EPON) fiber backhaul and an IEEE 802.11b/g/n/s wireless local area network (WLAN) mesh front end, which may be further upgraded by leveraging NG-PONs, notably $10+\mathrm{Gb}/\mathrm{s}$ TDM/WDM PONs, and gigabit-class IEEE 802.11ac very-high-throughput (VHT) WLAN technologies [38]. Thus, most FiWi access networks rely on low-cost data-centric optical fiber Ethernet (EPON) and wireless Ethernet (WLAN) technologies, which provide a couple of important benefits. First, economic considerations are expected to play an even more critical role in 5G networks than in the previous four generations. Second, today’s service providers have to cope with an unprecedented growth of mobile data traffic worldwide. Complementing 4G LTE-A HetNets with already widely deployed WiFi access points represents a key aspect of the strategy of today’s operators to offload mobile data traffic from their cellular networks, a technique known as WiFi offloading. FiWi access networks with a WLAN-based front end represent a promising approach to realize WiFi offloading in a cost-efficient manner.

Now, it is important to understand that, unlike LTE, WLANs use a distributed MAC protocol for arbitrating access to the wireless medium among stations. Specifically, the so-called distributed coordination function (DCF) typically deployed in WLANs may suffer from a seriously deteriorated throughput performance due to the propagation delay of the fiber backhaul. To see this, note that in WLANs a wireless source station starts a timer after each frame transmission and waits for the acknowledgment (ACK) from the wireless destination station. If the source station does not receive the ACK before the ACK timeout, it will resend the frame for a certain number of retransmission attempts. Clearly, one solution to compensate for the fiber propagation delay is to increase the ACK timeout. Note, however, that in DCF the ACK timeout must not exceed the DCF interframe space (DIFS), which prevents other stations from accessing the wireless medium and thus avoids collision with the ACK frame (in IEEE 802.11 WLAN specifications DIFS is set to 50 μs). Due to the ACK timeout, backhaul fiber can be deployed in WLAN-based FiWi networks only up to a maximum length. For instance, in a standard IEEE 802.11b WLAN network with a default ACK timeout value of 20 μs, the backhaul fiber length must be less than 1948 m to ensure the proper operation of DCF.

B. C-RAN: Cloud Versus Cloudlet

Clearly, the aforementioned limitations of WLAN-based FiWi access networks can be avoided by controlling access to the optical fiber and wireless media separately from each other, giving rise to so-called radio-and-fiber (R&F) networks [39]. R&F-based FiWi access networks may deploy a number of enabling optical and wireless technologies, including tunable lasers and receivers, colorless ONUs, as well as burst-mode laser drivers and receivers. In RoF networks, optical fiber is used as an analog or digital transmission medium between a central station and one or more remote antenna units with the central station in charge of controlling access to both optical and wireless media. In contrast, in R&F networks, access to the optical and wireless media is controlled separately by using in general two different MAC protocols in the optical and wireless media, with protocol translation taking place at their optical–wireless interface. As a consequence, wireless MAC frames do not have to travel along the backhaul fiber to be processed at any central control station but simply traverse their associated access point and remain in the WLAN. Access control is done locally inside the WLAN in a fully decentralized fashion and thus avoids the negative impact of fiber propagation delay. Note that in doing so, WLAN-based FiWi access networks of extended coverage can be built without imposing stringent limits on the length of the fiber backhaul. Recall that this holds only for distributed MAC protocols such as DCF but not for MAC protocols that deploy centralized polling and scheduling such as EPON and LTE. Thus, in a typical R&F-based FiWi access network consisting of a cascaded EPON backhaul and WLAN front end for WiFi offloading, the end-to-end coordination of both fiber and wireless network resources may be done by a co-DBA algorithm that uses the centralized IEEE 802.3ah multipoint control protocol (MPCP) for EPON and the decentralized DCF for WiFi, with MAC protocol translation taking place at the optical–wireless interface. Note that the decentralized nature of WLAN’s access protocol DCF is instrumental in realizing low-latency FiWi enhanced LTE-A HetNets, as explained in more detail in the subsequent section.

Next, let us consider an illustrative example to better understand the operation of co-DBA in FiWi access networks. A major MAC enhancement technique of next-generation WLANs is frame aggregation, which groups multiple wireless MAC frames into a single aggregate MAC protocol or service data unit for wireless transmission. In [40], the benefits of co-DBA were demonstrated by extending advanced frame aggregation techniques to EPON and their integrated operation across both optical and wireless segments. The proposed hierarchical frame aggregation techniques involve different aggregation layers, ranging from hop-by-hop to end-to-end aggregation of traffic between the OLT and wireless stations, and help improve the throughput-delay performance of R&F-based FiWi access networks for voice, video, and data traffic. For illustration, Fig. 7 depicts a FiWi access network consisting of a cascaded EPON backhaul and a WLAN-based mesh front end. The wireless mesh frontend comprises mesh portal points (MPPs) collocated with ONUs, intermediate mesh points (MPs), and mesh access points (MAPs), each serving associated wireless stations (STAs). The five different aggregation layers (L0–L4) shown in the upper part of Fig. 7 illustrate the possible operation modes of the co-DBA algorithm. Specifically, L0 applies frame aggregation only for traffic between the OLT and ONU-MPPs (as well as conventional ONUs without wireless extension), i.e., frame aggregation is used in the optical network segment separately from the wireless network segment. The remaining four aggregation layers (L1–L4) apply frame aggregation across the optical–wireless interface, thereby allowing for joint frame aggregation in both optical and wireless network segments, ranging from wireless single-hop MPs to all-wireless multi-hop MPs, MAPs, and STAs in an end-to-end fashion.

 

Fig. 7. Hierarchical frame aggregation involving different aggregation layers (L0–L4) across EPON backhaul and WLAN mesh front end. Adapted from [39]. © 2008 IEEE.

Download Full Size | PDF

In [41], we studied FiWi access networks in the context of both conventional clouds and emerging cloudlets, paying particular attention to the difference between R&F and traditional RoF networks. RoF networks were used in, for example, China Mobile’s C-RAN, which relies on a centralized cloud infrastructure and moves BBUs away from RRHs, intentionally rendering the RRHs as simple as possible without any processing and storage capabilities. Conversely, beside MAC protocol translation, the distributed processing and storage capabilities inherently built into R&F networks may be exploited for realizing a number of additional network functions. Therefore, in [41], we argued that R&F-based FiWi access networks may become the solution of choice in light of the aforementioned trends of future 5G mobile networks toward decentralization based on cloudlets and MEC. For completeness, however, we note that R&F and RoF technologies may also be used jointly for providing multi-tier cloud computing services, which accommodate both central cloud (e.g., C-RAN) and decentralized edge computing services over the same network infrastructure. For further details on multi-tier cloud computing in FiWi enhanced mobile networks, the interested reader is referred to [42–44].

IV. Low-Latency FiWi Enhanced LTE-A HetNets With Edge Intelligence

Recall from Section III.A that FiWi access networks provide a promising approach to offload mobile data from cellular networks by means of WiFi offloading. Recent backhaul-aware 4G studies have begun to investigate the performance-limiting impact of backhaul links in small-cell networks, although most of them did not take fiber link failures into account and assumed the reliability of the backhaul to be ideal (i.e., offering an availability of 100%).

To meet the URLLC requirements of 5G networks, in [45], we recently explored the performance gains obtained from enhancing coverage-centric 4G LTE-A HetNets with capacity-centric FiWi access networks based on low-cost, data-centric Ethernet NG-PON and gigabit-class WLAN technologies. Clearly, by unifying LTE-A HetNets and FiWi access networks, low-cost high-speed mobile data offloading is achievable via high-capacity fiber backhaul (e.g., IEEE 802.3av 10G-EPON) and gigabit-class WLAN that has been able to consistently provide data rates 100 times higher than cellular networks [46], thus helping reach the envisioned 1000-fold gains in area capacity and 10 Gb/s peak data rates of 5G.

In the following, we extend our concept of FiWi enhanced LTE-A HetNets in order to enable both local and non-local teleoperation by exploiting AI-based MEC capabilities. Note that neither teleoperation nor edge intelligence were addressed in [45].

A. Low-Latency FiWi Enhanced LTE-A HetNets

Figure 8 depicts the generic network architecture of FiWi enhanced LTE-A HetNets. The fiber backhaul consists of a TDM/WDM IEEE 802.3ah/av 1/10 Gb/s EPON with a typical fiber range of 20 km between the central OLT and remote ONUs. The EPON may comprise multiple stages, each stage separated by a wavelength-broadcasting splitter/combiner or wavelength multiplexer/demultiplexer. There are three different subsets of ONUs. An ONU may either serve fixed (wired) subscribers. Alternatively, an ONU may connect to either a cellular network base station (BS) or an IEEE 802.11n/ac/s WLAN MPP, giving rise to a collocated ONU-BS or ONU-MPP, respectively. Depending on positioning, a mobile user (MU) may communicate through the cellular network and/or WLAN mesh front end, which consists of ONU-MPPs, intermediate MPs, and MAPs. Note that connecting these three different sets of ONUs via a common shared EPON fiber backhaul infrastructure helps achieve the important goal of fixed-mobile convergence gain of today’s network operator strategy, as discussed in Section I.

 

Fig. 8. Local and non-local teleoperation in FiWi enhanced LTE-A HetNets with AI-based MEC capabilities.

Download Full Size | PDF

In [45], we presented various advanced fiber-lean backhaul redundancy strategies (not shown in Fig. 8), which can be used to realize a local Fx-FH with direct inter-ONU communication. Specifically, we considered the following three strategies: (i) interconnection fiber links between pairs of neighboring ONUs, (ii) small-scale fiber protection rings among multiple nearby ONUs, and (iii) wireless bypassing of backhaul fiber faults via the WLAN front end. Our results showed that the proposed localized protection techniques are instrumental in providing fixed wired and mobile users with highly fault-tolerant FiWi connectivity. Recall from Section I that Fx-FH solutions also help reduce latency by forming local clusters of ONUs as well as ONU-MPPs, thereby increasing the diversity of network connections. Our analytical results verified by recent comprehensive smartphone traces showed that the presented interconnection fiber, protection ring, and wireless protection techniques are able to keep the FiWi connectivity probability of MUs essentially flat for a wide range of EPON fiber-link failure probabilities while decreasing the average end-to-end delay to 1 ms for a wide range of traffic loads.

To better understand the reason behind the low delay performance of FiWi enhanced LTE-A HetNets, note that LTE systems themselves cannot guarantee low latency due to the fact that the transmission time interval is 1 ms. Thus, both uplink and downlink transmissions take at least 1 ms, translating into an end-to-end delay being lower bounded by 2 ms. In real-world deployment scenarios, the latency in LTE networks may increase by an order of magnitude. On the other hand, low-latency WiFi technology can bring 5G level of service today if the network is properly set up to mitigate interference, given that DCF does not impose inherent latency limitations per se in that it allows users to immediately access (after a short DIFS of 50 μs) the idle wireless channel in a decentralized manner [47].

In this work, unlike [45], which studied only conventional H2H communication between MUs, we investigate the potential and limits of coexistent teleoperation in FiWi enhanced LTE-A HetNets. Given the typical WiFi-only operation of state-of-the-art robots [29], HOs and TORs are assumed to communicate only via WLAN, as opposed to MUs who use dual-mode 4G/WiFi smartphones. Teleoperation is done either locally or non-locally, depending on the proximity of the involved HO and TOR, as illustrated in Fig. 8. In local teleoperation, the HO and corresponding TOR are associated with the same MAP and exchange their command and feedback samples through this MAP without traversing the fiber backhaul. Conversely, if HO and TOR are associated with different MAPs, non-local teleoperation is generally done by communicating via the backhaul EPON and central OLT. For simplicity, in this work we focus on the generic network architecture of FiWi enhanced LTE-A HetNets, shown in Fig. 8, without leveraging direct inter-ONU communication.

B. Edge Intelligence

Despite recent interest in exploiting machine learning for optical communications and networking, edge intelligence for enabling an immersive and transparent teleoperation experience for human operators has not been explored yet. In the following, we introduce machine learning at the edge of our considered communication network for realizing immersive and frictionless Tactile Internet experiences.

To realize edge intelligence, selected ONU-BSs/MPPs are equipped with AI-enhanced MEC servers. These servers rely on the computational capabilities of cloudlets collocated at the optical-wireless interface (see Fig. 8) to forecast delayed haptic samples in the feedback path. Towards this end, we deploy a type of parameterized artificial neural network (ANN) known as a multi-layer perceptron (MLP), which is capable of approximating any linear/nonlinear function to an arbitrary degree of accuracy [48]. Note that an MLP with $N_h$ hidden neurons represents a linear combination of $N_h$ parameterized nonlinear functions called neurons. Furthermore, note that a neuron is a nonlinear function $\mathcal{G}(·)$ of a linear combination of its input variables. In this work, the ANN is an MLP with $L$ input variables and one output variable. More specifically, $\mathrm{\Xi }$denotes the set of $L·N_h+N_h+1$ weights of the model, i.e., $\mathrm{\Xi }=\{c_{i,j}\text{:}i=1,\dots ,N_h,j=1,\dots ,L\}\cup$$\{c_j^{\prime }\text{:}j=0,1,\dots ,N_h\}$, which are estimated during the training phase. The MLP yields the following output:

Haptic feedback plays a crucial role in providing the HO with transparency, immersion, and togetherness with the remote teleoperation environment [33]. To do so, we present an edge sample forecast (ESF) module based on the aforementioned MLP to compensate for delayed haptic feedback samples by means of multiple-sample-ahead-of-time forecasting. As a result, the response time of the HO can be kept small, which in turn leads to a tighter togetherness with the remote TOR and an enhanced immersion. In a nutshell, our developed MLP-based ESF module forecasts the force samples in the feedback path in time. More specifically, instead of waiting for the force samples that are delayed by more than a given waiting deadline $T_{\mathrm{thr}}$, the module locally generates and delivers the forecast feedback samples to the HO. Let us refer to the feedback signal to be forecast as the target signal $X(·)$, i.e., the force feedback samples in our case. Our objective is to generate at any time instant $t$ a forecast sample denoted by ${\theta }^{*}$ for time instant $t_0=t-T_{\mathrm{thr}}$, whereby $T_{\mathrm{thr}}$ is the maximum period of time that the HO can wait until receiving the actual sample $\theta =X(t_0)$. More precisely, at any time $t$, if the sample for time instant $t_0$ is not received, a forecast sample is generated and immediately delivered to the HO. This procedure is repeated every 1 ms, which equals the typical intersample time of teleoperation systems. Note that the proposed MLP predicts $\theta$ from the past observations of the target signal. For a technically more detailed description of our proposed ESF module, we refer the interested reader to Appendix A.

To create our training dataset, we used the available 6-DoF teleoperation traces. We used MATLAB to build and train a one-hidden-layer ANN. Our training data set comprised 59,710 force feedback samples with the waiting deadline set to $T_{\mathrm{thr}}=1\mathrm{ms}$. We used the so-called Levenberg–Marquardt training method for adjusting the weights until a desired input/output relationship was obtained. Prior to simulations, we applied brute force for determining the optimal value of the number of neurons in the hidden layer, which led us to set it to 5. After training, we used a new data set comprising 1,000 samples (different from the training data set) to evaluate the performance of our proposed sample forecaster in terms of mean squared error between the actual and forecast samples. It is worthwhile to mention that once the training was complete, the ANN was run on the HO side to provide the HO with forecast samples. Moreover, we note that the processing/running delay of the developed ANN on the order of microseconds was relatively small compared to the communication-induced packet delays.

For completeness, we note that a one-hidden-layer MLP is also known as a universal approximator. We decided to use a one-hidden-layer MLP since it is simple (i.e., easy to implement and train) yet achieves an accuracy that is good enough to approximate a wide variety of linear and/or nonlinear functions. Besides longer training times, note that increasing the number of hidden layers in our considered one-hidden-layer MLP may result in over-fitting, which in turn may have a detrimental impact on its forecasting accuracy.

Note that in our considered FiWi enhanced LTE-A HetNets architecture in Fig. 8, all HOs and TORs are connected through a shared fiber backhaul whose fiber reach does not exceed the typical 20 km of an IEEE 802.3ah EPON or up to 100 km in case of long-reach PONs. The limited fiber reach keeps the propagation delay below 0.1 ms and 0.5 ms, respectively. Thus, in a conventional EPON and in long-reach PONs, the fiber propagation delay does not pose a challenge to meeting the 1 ms latency requirement of the Tactile Internet. However, an interesting question is how the 1-ms challenge of the Tactile Internet can be addressed for significantly larger geographical distances, e.g., connecting HOs in North America with TORs in Europe and/or Asia. This is where our proposed ESF module offers a potentially promising solution in that it decouples haptic feedback from the impact of extensive propagation delays, as typically encountered in wide area optical fiber networks. To see this, Fig. 9 illustrates our ESF module for the general case of a communication network with arbitrary propagation delays. The ESF module may be inserted at the edge of the communication network in close proximity to the HO. Rather than waiting for delayed haptic feedback samples that exceed the waiting deadline of 1 ms, the ESF module generates forecast samples and delivers them to the HO. Hence, the HO is enabled to perceive the remote task environment in time at a 1-ms granularity, resulting in a tighter togetherness, improved safety control, and increased reliability of the teleoperation systems. It should be noted, however, that a more rigorous experimental investigation would be needed to validate the viability of our proposed ESF module for real-world deployment scenarios with various wide area network propagation delays.

 

Fig. 9. Edge sample forecast (ESF) module at the edge of a general communication network with arbitrary propagation delays.

Download Full Size | PDF

Clearly, the capability of our proposed ESF module to enable HOs to perceive the remote task environment in time at a 1-ms granularity requires a sufficiently high forecasting accuracy of haptic feedback samples, as discussed in more detail in the subsequent section.

V. Results

We have extended our previous analytical frameworks in [38,45] to compute the average end-to-end delay and its distribution for local and non-local teleoperation with coexistent H2H traffic. For details on our extended analytical framework and its underlying assumptions we refer the interested reader to Appendix B.

In the following, we assume that MUs, HOs, and TORs are directly connected to their associated MPPs, i.e., MPPs serve as conventional WLAN access points (APs). By default, let us consider four ONU-APs, each with four associated MUs, whereby two MUs communicate with each other via their associated ONU-AP using an IEEE 802.11n WLAN (i.e., local H2H communications) while the two remaining MUs communicate with two uniformly randomly selected MUs associated with a different ONU-AP by using a backhaul IEEE 802.3ah 1 Gb/s EPON with a typical fiber range of 20 km (i.e., non-local H2H communications). Furthermore, let us consider four conventional ONUs, serving fixed (wired) subscribers who are all involved in non-local H2H communications among one another. The MUs and fixed subscribers generate background traffic at a mean rate of ${\lambda }_{\mathrm{BKGD}}$ and ${\alpha }_{\mathrm{PON}}·{\lambda }_{\mathrm{BKGD}}$, respectively. Note that ${\alpha }_{\mathrm{PON}}\ge 1$ is a traffic scaling factor for fixed subscribers who are directly connected to the backhaul EPON. Figure 10 depicts the average end-to-end delay of MUs versus mean background traffic rate ${\lambda }_{\mathrm{BKGD}}$ with different ${\alpha }_{\mathrm{PON}}\in \{1,50,100\}$ for both local and non-local H2H communications in FiWi enhanced LTE-A HetNets. The figure shows that an average end-to-end delay of ${10}^0=1\mathrm{ms}$ can be achieved for non-local H2H communications for a wide range of background traffic loads.

 

Fig. 10. Average end-to-end delay of mobile users (MUs) versus mean background traffic rate ${\lambda }_{\mathrm{BKGD}}$ (packets/s) for local and non-local H2H communications with different ${\alpha }_{\mathrm{PON}}\in \{1,50,100\}$.

Download Full Size | PDF

Next, we include teleoperation and investigate the interplay between Tactile Internet traffic and the above H2H background traffic. Towards this end, we consider the above scenario and replace two MUs with a pair of HO and TOR in the coverage area of each ONU-AP for local teleoperation with and without deadband coding in the command path. Specifically, we consider our findings on 6-DoF teleoperation in Fig. 6(a) and accordingly assume gamma and GP distributed haptic packet arrivals for $d_c\in \{0,0.01\%,0.02\%\}$ and $d_c=0.05\%$, respectively. Figure 11 depicts the average end-to-end delay of HOs versus mean background traffic rate ${\lambda }_{\mathrm{BKGD}}$ along with verifying trace-driven simulations based on our 6-DoF haptic traces and packetization procedure described in Section II.B.1. We observe from Fig. 11 that without deadband coding ($d_c=0$), the minimum achievable average end-to-end delay experienced by HOs equals 4.62 ms, and thus it misses the Tactile Internet target of 1 ms. However, note that this target can be achieved with deadband coding for increasing $d_c$. For illustration, Fig. 11 shows that we achieve a minimum average end-to-end delay of 1.18 ms for $d_c=0.05\%$. In addition to decreasing the latency of HOs, note that deadband coding also has a beneficial impact on the admissible background traffic load of MUs due to the reduced haptic packet rates. To see this, let us define the coding gain $G_{\text{coding}}$ as the difference between the maximum admissible throughput of MUs in teleoperation with and without deadband coding, while not violating a certain upper average end-to-end delay limit. For instance, for a given upper limit of 4.8 ms, a coding gain of $G_{\text{coding}}=1.42$ Mbps per MU can be achieved in our teleoperation scenario by increasing $d_c$ from 0 to 0.01%, as depicted in Fig. 11. Note that overall the presented analytical results and verifying trace-driven simulation results (shown with 95% confidence interval) match very well.

 

Fig. 11. Average end-to-end delay of human operators (HOs) versus mean background traffic rate ${\lambda }_{\mathrm{BKGD}}$ (packets/s) for local teleoperation with and without deadband coding in the command path for different $d_c\in \{0,0.01\%,0.02\%,0.05\%\}$ (${\alpha }_{\mathrm{PON}}=100$ fixed).

Download Full Size | PDF

Figure 12 provides useful insights into the upper end-to-end delay bounds by showing its cumulative distribution function (CDF) $F_{D_{LT(i\to j)}^{E2E}}(t)$ for the scenario of Fig. 11. Notably, we observe that for $d_c=0.05\%$ and a high rate ${\lambda }_{\mathrm{BKGD}}=20\mathrm{packets}/\mathrm{s}$ (top curve), the end-to-end delay stays below 2 ms with a probability as high as 0.8.

 

Fig. 12. End-to-end delay CDF $F_{D_{LT(i\to j)}^{E2E}}(t)$ of local teleoperation.

Download Full Size | PDF

To provide insight into the impact of different NG-PON backhaul infrastructures in the case of non-local teleoperation, Fig. 13 depicts the average end-to-end delay performance of HOs versus backhaul traffic scale factor ${\alpha }_{\mathrm{PON}}$of fixed subscribers with the mean background traffic rate set to ${\lambda }_{\mathrm{BKGD}}=20\mathrm{packets}/\mathrm{s}$. For comparison, we consider a conventional 1 Gbps EPON, a high-speed 10-Gbps EPON, and a WDM PON with $\mathrm{\Lambda }=2$ wavelength channels, each operating at 1 Gb/s. Note that for all three considered NG-PONs we include a conventional fiber reach of $l_{\mathrm{PON}}=20\mathrm{km}$ as well as its respective long-reach counterpart with an extended fiber reach of $l_{\mathrm{PON}}=100\mathrm{km}$. We observe from Fig. 13 that the use of deadband coding ($d_c=0.05\%$) is instrumental in lowering the average end-to-end delay below 10 ms for all NG-PON backhaul infrastructures under consideration. The figure also confirms previous findings (see Section I) that 10G PON and WDM technologies represent cost-effective solutions to support 5G low-latency applications over a wide range of backhaul traffic loads by sharing a common optical transport platform among fixed subscribers, MUs, and HOs.

 

Fig. 13. Average end-to-end delay of human operators (HOs) versus backhaul traffic scale factor ${\alpha }_{\mathrm{PON}}$ of fixed subscribers (${\lambda }_{\mathrm{BKGD}}=20\text{packets}/\mathrm{s}$ fixed) for non-local teleoperation across different NG-PON backhaul infrastructures.

Download Full Size | PDF

We have seen in the results above that deadband coding is effective in decreasing the average end-to-end delay by reducing the haptic packet rate. Nevertheless, some haptic packets may still experience an instantaneous delay that exceeds the desired waiting deadline on the order of 1 ms until their reception due to varying traffic conditions and MAC layer queueing times. To ensure that the HO receives expected haptic packets before the deadline, our proposed MLP-based ESF module may be used as a complementary technique to deadband coding in the feedback path. Figure 14 compares the forecasting accuracy of our proposed MLP-based ESF scheme with a naive ESF scheme, where the forecast sample is simply set to the last received sample. In our simulation, we used our 6-DoF teleoperation traces to train a one-hidden-layer MLP by using 59,710 force feedback samples with the waiting deadline set to $T_{\mathrm{thr}}=1\mathrm{ms}$. Figure 14 clearly shows the superior forecasting accuracy of our proposed MLP-based ESF scheme in terms of mean squared error over a wide range of ${\lambda }_{\mathrm{BKGD}}$ for both local and non-local teleoperation scenarios, whereby a low mean squared error is achievable in the former scenario. Specifically, for non-local teleoperation, our MLP-based ESF scheme decreases the mean squared error from roughly 0.9 to $0.65\times {10}^{-3}$, translating into an improvement of 27.8%. For local teleoperation, it is able to keep the mean squared error close to zero between 0.006 and $0.007\times {10}^{-3}$ at low to medium background traffic load ${\lambda }_{\mathrm{BKGD}}$. Note that the observed performance improvement is due to the relatively high autocorrelation in the haptic feedback samples that allows our proposed MLP-based ESF module to achieve a more accurate forecast compared to that of the naive ESF scheme.

 

Fig. 14. Comparison of forecasting accuracy between proposed MLP-based and naive ESF schemes for local and non-local teleoperation without deadband coding in the feedback path ($d_f=0$).

Download Full Size | PDF

VI. Tactile Internet: Where Do We Go From Here?

The Internet has been constantly evolving from the mobile Internet to the emerging IoT and future Tactile Internet. Similarly, the capabilities of future 5G networks will extend far beyond those of previous generations of mobile communication. In [49], we recently outlined some research ideas that help tap into the full potential of the Tactile Internet. More specifically, we shed some light on various concepts that will be instrumental in creating new ideas to facilitate local human–machine coactivity clusters by completely decentralizing edge computing via emerging Ethereum blockchain technologies.

The next-generation Internet powered by decentralized blockchain technology is ushering in a new era [50]. A blockchain of particular interest is Ethereum, which enables the development of decentralized applications that are not limited to cryptocurrencies, a capability that Bitcoin lacked [51]. A salient feature of Ethereum that cannot be found in the Bitcoin blockchain is the decentralized autonomous organization (DAO). Unlike AI-based agents that are completely autonomous, a DAO still requires heavy involvement from humans. For illustration, Fig. 15 shows a quadrant chart that classifies DAOs, AI, robots, and traditional organizations with regard to automation and humans involved at their edges and center. Note that while most technologies tend to automate workers on the periphery doing menial tasks, Ethereum automates the center.

 

Fig. 15. DAOs versus AI, robots, and traditional organizations: automation and humans involved at their edges and center (source: Ethereum Blog).

Download Full Size | PDF

Despite recent progress on realizing the blockchain IoT (B-IoT), at present it is yet unclear in many ways how Ethereum blockchain technologies, in particular the DAO, may be leveraged to realize future techno-social systems such as the Tactile Internet that by design still require heavy involvement from humans at the network edge instead of automating them away [52]. For a deep discussion on the human-agent-robot teamwork (HART) concept and how the relative work capabilities of humans and robots affect the task completion time, please see [30].

VII. Conclusions

We have seen that there is a significant overlap among 5G, IoT, and the Tactile Internet in that they share various important design goals, including very low latency, ultra-high reliability, and integration of data-centric technologies. In this paper, we described how FiWi enhanced LTE-A HetNets leveraging low-cost data-centric EPON and WiFi technologies for fiber backhaul sharing and WiFi offloading may help realize not only the aforementioned shared design goals but also the key attributes of end-to-end co-DBA of both PON and wireless network resources, decentralization, and edge intelligence in support of future 5G low-latency applications over a common optical transport platform.

Our focus was on the emerging Tactile Internet as one of the most interesting 5G low-latency applications for creating novel immersive experiences. We reviewed the HITL-centric design principles that add a new dimension to the human-to-machine interaction via the Internet and set the Tactile Internet aside from the more machine-centric IoT. Exploiting the human perception of haptics to reduce the haptic packet rate by means of deadband coding, we derived haptic traffic models from teleoperation experiments. Our haptic trace analysis showed that assuming Tactile Internet traffic to be Pareto distributed was not valid for the analyzed traffic, while assuming it to be Poisson traffic was valid only in a special case. In general, we observed that command and feedback paths of teleoperation systems can be jointly modeled by generalized Pareto, gamma, or deterministic packet interarrival time distributions, depending on the given value of the respective deadband parameters.

In our comparison with C-RAN, we elaborated on the importance of the decentralized nature of WLAN’s access protocol DCF to realize low-latency FiWi enhanced LTE-A HetNets. Furthermore, by exploiting their inherent distributed processing and storage capabilities, we investigated the potential of enabling immersive teleoperation experiences for human operators by introducing machine learning at the optical–wireless interface of FiWi enhanced LTE-A HetNets. Our proposed MLP-based ESF module compensates for delayed haptic feedback samples by means of multiple-sample-ahead-of-time forecasting for a tighter togetherness, improved safety control, and increased reliability. Future work will investigate the applicability of this technique for networks with arbitrary delays.

Appendix A: Edge Sample Forecast (ESF)

The objective is to generate at any time $t$ a forecast sample ${\theta }^{*}$ that corresponds to time instant $t_0=t-T_{\mathrm{thr}}$, where $T_{\mathrm{thr}}$ is the waiting deadline until which the HO can wait to receive the actual sample $\theta =X(t_0)$. Let $\mathcal{S},\mathcal{T}\in {\mathbb{R}}^K$ denote the last $K$ samples $\{s_1,s_2,\dots ,s_K\}$ at time stamps $\{t_1,t_2,\dots ,t_K\}$. Note that $\mathcal{S},\mathcal{T}$ are used to forecast sample ${\theta }^{*}$ at any time instant $t_0>t_K$. The feedback sample is forecast by Algorithm 1 with input $\mathcal{S},\mathcal{T},t_0,\mathrm{\Xi }$ and output ${\theta }^{*}$. We define $\delta$ as the intersample time step in our sample forecaster and set it to $1/F_s$, where $F_s$ denotes the sampling frequency of 1 kHz (line 7 in Algorithm 1). To align the received samples in time, we call the SAMPLE_ALIGNER() procedure (Algorithm 2) with input $\mathcal{S}\in {\mathbb{R}}^K$, $\mathcal{T}\in {\mathbb{R}}^K$, and $\delta$ and the output consisting of the aligned sample set ${\mathcal{S}}^{\delta }\in {\mathbb{R}}^L$ and the time stamp set to ${\mathcal{T}}^{\delta }\in {\mathbb{R}}^L$.

Next, we calculate the forecast horizon $\mathrm{\Delta }$ at time $t$, which denotes the estimated number of samples during time interval ${\mathcal{T}}^{\delta }(L)$ between the last observed sample and target time $t_0$ (line 7 in Algorithm 1). Our objective is to forecast sample set $\mathbf{\Theta }=\{{\theta }_1,{\theta }_2,\dots ,{\theta }_{\mathrm{\Delta }}\}$ for time stamp set $\{t_1^{*},t_2^{*},\dots ,t_{\mathrm{\Delta }}^{*}\}$ to finally estimate sample ${\theta }^{*}$ at time $t_0$. Specifically, sample ${\theta }_i\in \mathbf{\Theta }$ is forecast by feeding our MLP with input vector ${\mathcal{A}}_{i-1}\in {\mathbb{R}}^L$, where ${\mathcal{A}}_0=(s_1^{\delta },\dots ,s_L^{\delta })\in {\mathbb{R}}^L$ and ${\mathcal{A}}_i=({\mathcal{A}}_{i-1}(2),{\mathcal{A}}_{i-1}(3),\dots ,{\mathcal{A}}_{i-1}(L),{\theta }_i)$, i.e., each sample is forecast based on the preceding $L$ samples. To further improve the forecasting accuracy, we estimate ${\theta }^{*}$ by performing a two-point linear interpolation between ${(t_{\mathrm{\Delta }-1}^{*},{\theta }_{\mathrm{\Delta }-1})}^T$ and ${(t_{\mathrm{\Delta }}^{*},{\theta }_{\mathrm{\Delta }})}^T$ (line 7 in Algorithm 1).

Algorithm 1. Edge Sample Forecast

View Table | View all tables in this article

Algorithm 2. SAMPLE_ALIGNER()

View Table | View all tables in this article

Appendix B: End-to-End Delay Analysis

In the following, we develop our analytical framework to compute the average end-to-end delay and its distribution for local and non-local teleoperation with coexistent H2H traffic. We use the term “WiFi user” for all MUs, HOs, and TORs within the coverage area of an ONU-AP (i.e., single-hop wireless connection). Similar to [38,45], the WiFi channel access time arbitrated by IEEE 802.11 DCF is assumed to be exponentially distributed. We model each WiFi user as a GI/M/1 queue to account for the different packet interarrival time distributions under consideration. Let random variable $D$ denote the delay experienced by any packet generated by a WiFi user, where $D$ comprises queueing delay $D_Q$ and service time $D_S$. Suppose that the interarrival times are mutually i.i.d. random variables with distribution function $G(t)$. We define ${\pi }_j$ as the stationary state probability that an arriving packet finds $j$ packets in the system (i.e., queue and service). For a GI/M/I queue, the stationary state probabilities ${\pi }_j$ have a geometric distribution given by

(B1)$${\pi }_j=(1-\omega ){\omega }^j,$$

whereby $\omega$ is obtained by solving $\omega =\mathrm{\varphi }(z){|}_{z=(1-\omega )\mu }$, where $\mathrm{\varphi }(z)$ is the Laplace–Stieltjes transform of $G(x)$ and $\mu$ denotes the service rate, which is equal to $1/\mathbb{E}[D_S]$. For a GI/M/1 queue, the CDFs of $D_Q$ and $D_s$ are calculated as

(B2)$$F_{D_Q}(t)=P(D_Q\le t)=1-\omega e^{-(1-\omega )\mu t},$$

(B3)$$F_{D_S}(t)=P(D_S\le t)=1-e^{-\mu t},$$

respectively, which are used to calculate the CDF of $D=D_Q+D_S$ at a WiFi user as follows:

(B4)$$F_D(t)=P(D\le t)={\int }_0^t{F}_{D_S}(t-u)\mathrm{d}{F}_{D_Q}(u).$$

To compute the service rate $\mu$ in Eq. (B3), we defined the two-dimensional Markov process $(s(t),b(t))$ under unsaturated non-Poisson traffic conditions and estimated the average service time $\mathbb{E}(D_S)$ in a WLAN using IEEE 802.11 DCF for access control, whereby $b(t)$ and $s(t)$ denote the random backoff counter and size of the contention window at time $t$, respectively. Let $W_i$ denote the contention window size at back-off stage $i$. After finding the stationary distributions

$$b_{i,k}=\underset{k\to \infty }{\lim }P(s(t)=i,b(t)=k),\forall k\in [0,W_i-1],i\in [0,m],$$

the probability $\tau$ that a WiFi user attempts to transmit in a given time slot is obtained as

(B5)$$\tau =\sum _{i=0}^m{b}_{i,0}=\frac{\frac{2(1-2P_f)q_2}{2(1-q_1)(1-P_f)(1-2P_f)}}{\frac{q_2[(W_0+1)(1-2P_f)+W_0{P}_f(1-{(2P_f)}^m)]}{2(1-q_1)(1-P_f)(1-2P_f)}+1},$$

where $q_1$ and $q_2$ are equal to $1-{\pi }_0$ and $\lambda /\mu$, respectively.

Given the probability $p_e$ of an erroneous transmission and the probability $p_c$ {given in Eq. (23) of [38]} of a collision, the probability of a failed transmission attempt $P_f$ is calculated using Eq. (24) of [38]. We then obtain $\mathbb{E}(D_S)$ as

(B6)$$\begin{gathered} \mathbb{E}(D_S)=\frac{1}{\mu }=\sum _{k=0}^{\infty }{p}_e^k(1-p_e)[\sum _{j=0}^{\infty }{p}_c^j(1-p_c)· \\ (\left(\sum _{b=0}^{k+j}\frac{2^{\min (b,m)}{W}_0-1}{2}{E}_s\right)+j{T}_c+k{T}_e+T_s\left)\right], \end{gathered}$$

where $E_s$ is given in Eq. (30) of [38]. To obtain the steady-state values of $q_1$, $q_2$, $P_f$, $\tau$, and $\mu$, we numerically solve the system of nonlinear equations (B6), Eq. (23) of [38], (B5), and (B1).

To compute the average end-to-end delay of non-local teleoperation, we need to estimate the average delay of the backhaul EPON in the upstream ${\overline{D}}_{\mathrm{PON}}^u$ and downstream ${\overline{D}}_{\mathrm{PON}}^d$ directions. ${\overline{D}}_{\mathrm{PON}}^u$ and ${\overline{D}}_{\mathrm{PON}}^d$ are given by $\mathrm{\varphi }({\rho }^u,\overline{L},{\varsigma }_L^2,c_{\mathrm{PON}})+\overline{L}/c_{\mathrm{PON}}+2{\tau }_{\mathrm{PON}}\frac{2-{\rho }^u}{1-{\rho }^u}-B^u$ and $\mathrm{\varphi }({\rho }^d,\overline{L},{\varsigma }_L^2,c_{\mathrm{PON}})+\overline{L}/c_{\mathrm{PON}}+{\tau }_{\mathrm{PON}}-B^d$, respectively, whereby ${\rho }^u$ is the traffic intensity in the upstream direction, ${\rho }^d$ is the traffic intensity in the downstream direction, ${\tau }_{\mathrm{PON}}$ is the propagation delay between ONUs and the OLT, $c_{\mathrm{PON}}$is the EPON data rate, and $\mathrm{\varphi }(·)$ denotes the well-known Pollaczek–Khintchine formula. $B^u$ and $B^d$ are obtained as $\mathrm{\varphi }(\frac{\overline{L}}{\mathrm{\Lambda }{c}_{\mathrm{PON}}}\sum _{i=1}^O\sum _{q=1}^O{\mathrm{\Gamma }}_{iq}^{\mathrm{PON}},\overline{L},{\varsigma }_L^2,c_{\mathrm{PON}})$, where $O$ is the number of ONUs and ${\mathrm{\Gamma }}_{iq}^{\mathrm{PON}}$ is the traffic emanating from ${\mathrm{ONU}}_i$ to ${\mathrm{ONU}}_q$; $\mathrm{\Lambda }$ denotes the number of wavelength channels in the WDM PON.

Acknowledgment

This work was supported by NSERC Discovery Grant 2016-04521 and the FRQNT B2X doctoral scholarship programme.

Footnotes

1	OFC/NFOEC workshop on “Post NG-PON2: Is it More About Capacity or Something Else,” 2013.
2	Phantom Omni is a widely used HSI device that enables HOs to interact with and manipulate objects by adding 3D navigation to a broad range of applications, e.g., games, entertainment, visualization, among others.

References

1. M. Maier and N. Ghazisaidi, FiWi Access Networks, Cambridge, UK: Cambridge University, 2012.

2. M. A. Lema, A. Laya, T. Mahmoodi, M. Cuevas, J. Sachs, J. Markendahl, and M. Dohler, “Business case and technology analysis for 5G low latency applications,” IEEE Access, vol. 5, pp. 5917–5935, 2017. [CrossRef]  

3. P. Green, “Progress in optical networking,” IEEE Commun. Mag., vol. 39, no. 1, pp. 54–61, Jan. 2001. [CrossRef]  

4. J.-I. Kani, F. Bourgart, A. Cui, A. Rafel, M. Campbell, R. Davey, and S. Rodrigues, “Next-generation PON—Part I: Technology roadmap and general requirements,” IEEE Commun. Mag., vol. 47, no. 11, pp. 43–49, Nov. 2009. [CrossRef]  

5. M. Maier, “The escape of Sisyphus or what “Post NG-PON2” should do apart from neverending capacity upgrades,” Photonics, vol. 1, no. 1, pp. 47–66, Mar. 2014, special issue on All Optical Networks for Communications. [CrossRef]  

6. T. Biermann, L. Scalia, C. Choi, W. Kellerer, and H. Karl, “How backhaul networks influence the feasibility of coordinated multipoint in cellular networks,” IEEE Commun. Mag., vol. 51, no. 8, pp. 168–176, Aug. 2013. [CrossRef]  

7. T. Pfeiffer, “Can PON technologies accelerate 5G deployments?” in Conf. on Optical Network Design and Modelling (ONDM), Workshop on Optical Technologies in the 5G Era, Dublin, Ireland, May 2018.

8. “5G wireless fronthaul requirements in a PON context,” ITU-T Recommendation G.Sup.5GP, Oct. 2018.

9. J. Li and J. Chen, “Passive optical network based mobile backhaul enabling ultra-low latency for communications among base stations,” J. Opt. Commun. Netw., vol. 9, no. 10, pp. 855–863, Oct. 2017. [CrossRef]  

10. C. Ranaweera, M. G. C. Resende, K. Reichmann, P. Iannone, P. Henry, B.-J. Kim, P. Magill, K. N. Oikonomou, R. K. Sinha, and S. Woodward, “Design and optimization of fiber optic small-cell backhaul based on an existing fiber-to-the-node residential access network,” IEEE Commun. Mag., vol. 51, no. 9, pp. 62–69, Sept. 2013. [CrossRef]  

11. C. S. Ranaweera, P. P. Iannone, K. N. Oikonomou, K. C. Reichmann, and R. K. Sinha, “Design of cost-optimal passive optical networks for small cell backhaul using installed fibers [Invited],” J. Opt. Commun. Netw., vol. 5, no. 10, pp. A230–A239, Oct. 2013. [CrossRef]  

12. H. Kim, “RoF-based optical fronthaul technology for 5G and beyond,” in Optical Fiber Communications Conf. and Exposition (OFC), San Diego, California, Mar. 2018, pp. 1–3.

13. L. Velasco, A. Castro, A. Asensio, M. Ruiz, G. Liu, C. Qin, R. Proietti, and S. J. B. Yoo, “Meeting the requirements to deploy cloud RAN over optical networks,” J. Opt. Commun. Netw., vol. 9, no. 3, pp. B22–B32, Mar. 2017. [CrossRef]  

14. S. Zhou, X. Liu, F. Effenberger, and J. Chao, “Low-latency high-efficiency mobile fronthaul with TDM-PON (mobile PON),” J. Opt. Commun. Netw., vol. 10, no. 1, pp. A20–A26, Jan. 2018. [CrossRef]  

15. G. O. Pérez, J. A. Hernández, and D. Larrabeiti, “Fronthaul network modeling and dimensioning meeting ultra-low latency requirements for 5G,” J. Opt. Commun. Netw., vol. 10, no. 6, pp. 573–581, June 2018. [CrossRef]  

16. B. P. Rimal, D. Pham Van, and M. Maier, “Mobile edge computing empowered fiber-wireless access networks in the 5G era,” IEEE Commun. Mag., vol. 55, no. 2, pp. 192–200, Feb. 2017. [CrossRef]  

17. M. Satyanarayanan, “The emergence of edge computing,” IEEE Comput., vol. 50, no. 1, pp. 30–39, Jan. 2017. [CrossRef]  

18. F. Boccardi, R. W. Heath Jr., A. Lozano, T. L. Marzetta, and P. Popovski, “Five disruptive technology directions for 5G,” IEEE Commun. Mag., vol. 52, no. 2, pp. 74–80, Feb. 2014. [CrossRef]  

19. J. G. Andrews, S. Buzzi, W. Choi, S. V. Hanley, A. Lozano, A. C. K. Soong, and J. C. Zhang, “What will 5G be?” IEEE J. Sel. Areas Commun., vol. 32, no. 6, pp. 1065–1082, June 2014. [CrossRef]  

20. B. Skubic, M. Fiorani, S. Tombaz, A. Furuskär, J. Mårtensson, and P. Monti, “Optical transport solutions for 5G fixed wireless access [Invited],” J. Opt. Commun. Netw., vol. 9, no. 9, pp. D10–D18, Sept. 2017. [CrossRef]  

21. C. Behrens, S. Krauss, E. Weis, and D. Breuer, “Technologies for convergence of fixed and mobile access: an operator’s perspective [Invited],” J. Opt. Commun. Netw., vol. 10, no. 1, pp. A37–A42, Jan. 2018. [CrossRef]  

22. T. Taleb, K. Samdanis, B. Mada, H. Flinck, S. Dutta, and D. Sabella, “On multi-access edge computing: a survey of the emerging 5G network edge cloud architecture and orchestration,” IEEE Commun. Surv. Tutorials, vol. 19, no. 3, pp. 1657–1681, 2017. [CrossRef]  

23. G. P. Fettweis, “The tactile internet: applications and challenges,” IEEE Veh. Technol. Mag., vol. 9, no. 1, pp 64–70, Mar. 2014. [CrossRef]  

24. “The tactile internet,” ITU-T Technology Watch Report, Aug. 2014.

25. M. Simsek, A. Aijaz, M. Dohler, J. Sachs, and G. Fettweis, “5G-enabled Tactile Internet,” IEEE J. Sel. Areas Commun., vol. 34, no. 3, pp. 460–473, Mar. 2016. [CrossRef]  

26. A. Aijaz, M. Dohler, A. H. Aghvami, V. Friderikos, and M. Frodigh, “Realizing the Tactile Internet: haptic communications over next generation 5G cellular networks,” IEEE Wireless Commun., vol. 24, no. 2, pp. 82–89, Apr. 2017. [CrossRef]  

27. J. G. Andrews, “Seven ways that HetNets are a cellular paradigm shift,” IEEE Commun. Mag., vol. 51, no. 3, pp. 136–144, Mar. 2013. [CrossRef]  

28. A. Aijaz, Z. Dawy, N. Pappas, M. Simsek, S. Oteafy, and O. Holland, “Toward a Tactile Internet reference architecture: vision and progress of the IEEE P1918.1 Standard,” arXiv:1807.11915 (2018).

29. M. Maier, M. Chowdhury, B. P. Rimal, and D. Pham Van, “The tactile internet: vision, recent progress, and open challenges,” IEEE Commun. Mag., vol. 54, no. 5, pp. 138–145, May 2016. [CrossRef]  

30. M. Maier, A. Ebrahimzadeh, and M. Chowdhury, “The Tactile Internet: automation or augmentation of the human?” IEEE Access, vol. 6, pp. 41607–41618, July 2018. [CrossRef]  

31. K. Antonakoglou, X. Xu, E. Steinbach, T. Mahmoodi, and M. Dohler, “Towards haptic communications over the 5G Tactile Internet,” IEEE Commun. Surv. Tutorials, vol. 20, pp. 3034–3059, June 2018. [CrossRef]  

32. E. Weber, Die Lehre vom Tastsinn und Gemeingefuehl, auf Versuche gegruendet, London, UK: Verlag Friedrich Vieweg und Sohn, 1978.

33. E. Steinbach, S. Hirche, M. Ernst, F. Brandi, R. Chaudhari, J. Kammerl, and I. Vittorias, “Haptic communications,” Proc. IEEE, vol. 100, no. 4, pp. 937–956, Apr. 2012. [CrossRef]  

34. E. Wong, M. P. I. Dias, and L. Ruan, “Predictive resource allocation for Tactile Internet capable passive optical LANs,” J. Lightwave Technol., vol. 35, no. 13, pp. 2629–2641, July 2017. [CrossRef]  

35. L. Meli, C. Pacchierotti, and D. Prattichizzo, “Experimental evaluation of magnified haptic feedback for robot-assisted needle insertion and palpation,” Int. J. Med. Robot. Comput. Assist. Surg., vol. 13, no. 4, pp. e1809, Feb. 2017. [CrossRef]  

36. X. Xu, C. Schuwerk, B. Cizmeci, and E. Steinbach, “Energy prediction for teleoperation systems that combine the time domain passivity approach with perceptual deadband-based haptic data reduction,” IEEE Trans. Haptics, vol. 9, no. 4, pp. 560–573, Oct. –Dec. 2016. [CrossRef]  

37. A. D. Hossain, M. Ummy, A. Hossain, and M. Kouar, “Revisiting FiWi: on the merits of a distributed upstream resource allocation scheme,” J. Opt. Commun. Netw., vol. 9, no. 9, pp. 773–781, Sept. 2017. [CrossRef]  

38. F. Aurzada, M. Lévesque, M. Maier, and M. Reisslein, “FiWi access networks based on next-generation PON and gigabit-class WLAN technologies: a capacity and delay analysis,” IEEE/ACM Trans. Netw., vol. 22, no. 4, pp. 1176–1189, Aug. 2014. [CrossRef]  

39. M. Maier, N. Ghazisaidi, and M. Reisslein, “The audacity of fiber-wireless (FiWi) networks (Invited Paper),” in ICST Int. Conf. on Access Networks (AccessNets), Las Vegas, Nevada, Oct. 2008, pp. 1–10.

40. N. Ghazisaidi and M. Maier, “Hierarchical frame aggregation techniques for hybrid fiber-wireless access networks,” IEEE Commun. Mag., vol. 49, no. 9, pp. 64–73, Sept. 2011. [CrossRef]  

41. M. Maier and B. P. Rimal, “The audacity of fiber-wireless (FiWi) networks: revisited for clouds and cloudlets (invited paper),” China Commun., vol. 12, no. 8, pp. 33–45, Aug. 2015. [CrossRef]  

42. B. P. Rimal, D. Pham Van, and M. Maier, “Cloudlet enhanced fiber-wireless access networks for mobile-edge computing,” IEEE Trans. Wireless Commun., vol. 16, no. 6, pp. 3601–3618, June 2017. [CrossRef]  

43. B. P. Rimal, D. Pham Van, and M. Maier, “Mobile-edge computing versus centralized cloud computing over a converged FiWi access network,” IEEE Trans. Netw. Serv. Manage., vol. 14, no. 3, pp. 498–513, Sept. 2017. [CrossRef]  

44. B. P. Rimal, M. Maier, and M. Satyanarayanan, “Experimental testbed for edge computing in fiber-wireless broadband access networks,” IEEE Commun. Mag., vol. 56, no. 8, pp. 160–167, Aug. 2018. [CrossRef]  

45. H. Beyranvand, M. Lévesque, M. Maier, J. A. Salehi, C. Verikoukis, and D. Tipper, “Toward 5G: FiWi enhanced LTE-A HetNets with reliable low-latency fiber backhaul sharing and WiFi offloading,” IEEE/ACM Trans. Netw., vol. 25, no. 2, pp. 690–707, Apr. 2017. [CrossRef]  

46. G. Fettweis and S. Alamouti, “5G: personal mobile Internet beyond what cellular did to telephony,” IEEE Commun. Mag., vol. 52, no. 2, pp. 140–145, Feb. 2014. [CrossRef]  

47. Aptilo Networks, “Why wait for 5G? Carrier Wi-Fi is here today,” Dec. 2016 [Online]. Available: www.wifinowevents.com.

48. K. Hornik, M. Stinchcombe, and H. White, “Multilayer feedforward networks are universal approximators,” Neural Netw., vol. 2, no. 5, pp. 359–366, 1989. [CrossRef]  

49. M. Maier, “The Tactile Internet: where do we go from here? (invited paper),” in IEEE/OSA/SPIE Asia Communications and Photonics (ACP) Conf., Hangzhou, China, Oct. 2018.

50. M. E. Peck, “Blockchains: how they work and why they’ll change the world,” IEEE Spectrum, vol. 54, no. 10, pp. 26–35, Oct. 2017. [CrossRef]  

51. V. Buterin, “A next-generation smart contract and decentralized application platform,” Ethereum White Paper [Online]. Available: www.ethereum.org.

52. R. Beck, “Beyond bitcoin: the rise of blockchain world,” IEEE Comput., vol. 51, no. 2, pp. 54–58, Feb. 2018. [CrossRef]  

Martin Maier is a full professor with the Institut National de la Recherche Scientifique (INRS), Montréal, Canada. He was educated at the Technical University of Berlin, Germany, and received M.Sc. and Ph.D. degrees (both with distinctions) in 1998 and 2003, respectively. In the summer of 2003 he was a postdoc fellow at the Massachusetts Institute of Technology (MIT), Cambridge. He was a visiting professor at Stanford University, Stanford, from October 2006 through March 2007. Furthermore, he was a co-recipient of the 2009 IEEE Communications Society Best Tutorial Paper Award. He was a Marie Curie IIF Fellow of the European Commission from March 2014 through February 2015. In March 2017, he received the Friedrich Wilhelm Bessel Research Award from the Alexander von Humboldt (AvH) Foundation in recognition of his accomplishments in research on FiWi enhanced networks. In May 2017, he was named one of the three most promising scientists in the category “Contribution to a better society” of the Marie Skłodowska-Curie Actions (MSCA) 2017 Prize Award of the European Commission. He is the founder and creative director of the Optical Zeitgeist Laboratory (www.zeitgeistlab.ca).

Amin Ebrahimzadeh received his B.Sc. and M.Sc. degrees in electrical engineering from the University of Tabriz in 2009 and 2011, respectively. From 2011 to 2015, he was with the Sahand University of Technology (SUT), Iran. He is currently pursuing his Ph.D. in the Optical Zeitgeist Laboratory at the Institut National de la Recherche Scientifique (INRS), Montreal, QC, Canada. His research interests include fiber-wireless networks, Tactile Internet, teleoperation, artificial intelligence enhanced mobile edge-computing, and multi-robot task allocation.