1. Introduction

Intra-Datacenter (DC) networks are undergoing a paradigm shift driven by the growing demands, diversity of services, and spread of server virtualization trends. This condition has fueled a substantial increase in the intra-DC network traffic which according to recent estimates would shortly account for 75% of the total DC traffic [1]. Conventionally DC networks are hierarchical networks operated by relying on Electrical Packet Switching (EPS), whereas more recently enhanced DC network architectures such as the leaf-spine [2] have been replacing the traditional DC fat-tree, but still such EPS-based networks are facing an increasing difficulty in fulfilling the emerging DC demands, besides suffering a critical scalability issue. On the other hand, introducing optical switching technologies into DC networks is a promising alternative that has attracted a wide research interest [3–5 ]. In this regard, we have recently proposed a highly-scalable photonic DC network that has a flat architecture and Torus topology as shown in Fig. 1 .

 

Fig. 1 The structure of HOPR and HOPR-based DC network.

Download Full Size | PDF

The new network strongly supports service diversity where multiple transmission schemes including optical packet switching (OPS), optical circuit switching (OCS), and virtual optical circuit switching (V-OCS) are all available for deployment on the same hardware platform and at the same wavelength [6]. This DC network is enabled by the hybrid optoelectronic router (HOPR) illustrated in Fig. 1, in which each HOPR is connected to the neighboring units through 100-Gbps (25-Gbps × 4λs) links and to a group of top-of-rack (ToR) switches through 10-GbE links. The 100-Gbps HOPR is an upgrade for the previously prototyped 10-Gbps HOPR [7] in which a new set of enabling technologies have been introduced [8]. More details about the operation of the new DC network can be found in [9]. The structure of HOPR can be roughly divided into two main parts; on one hand an optical packet switch comprising label processors (LPs), optical switch fabric, and optical fiber delay line (FDL), and on the other hand an optoelectronic shared buffer that acts as an aggregation switch converting the ToRs’ 10-GbE packets into 100-Gbps burst-mode optical packets that are in turn forwarded to their destinations in the optical domain. Either the packet is incoming into HOPR or generated by the shared buffer at the same HOPR unit, its label is handled by an LP, where the LP’s role is threefold including the recognition of packet destination, performing arbitration for contention resolution, and the generation of corresponding control signals for the optical switch.

Different approaches for optical packet labeling have been proposed [10–12 ]. Owing to the difficulty of processing labels with ultrafast bits, some proposals were made for labels and payloads with different formats. However as bits with dissimilar formats experience different effects after transmission, this choice is unfavorable due to the constraints added to the full system. Another important feature to consider in a labeling scheme is the maximum number of distinguishable labels that can be supported. We use a time-serial label that precedes the packet payload i.e., header, with the same format as the payload and of the same wavelength. Generally optical packets would undergo multi-hop transmission in the DC network, and thus scaling out the network implies the necessity of minimizing the node’s latency as well as its power consumption. One way for realizing the essential power reduction at the LP is to fully exploit the label’s feature of having a short duration. When using 100-Gbps optical packets, the standard Ethernet packet with maximum size is around 120 ns, whereas the label that has to be recognized by the LP is much shorter. Thus by limiting the LP’s power consumption to the short time interval in which the label exists; the total consumed power can be significantly reduced. In this paper, we are adopting this approach to realize a novel LP subsystem which has been enabled by three key achievements; 1) a novel optical trigger pulse generator, that consumes power for a very short duration upon packet arrival, 2) an enhanced optoelectronic converter that operates with optical trigger pulses of low energy by employing a modified conversion scheme, 3) a new packaging method for supplying the optical triggers through the backside of the converter chip so as to double the responsivity of its integrated photodetectors.

2. The structure of new label processor

Our approach for label processing [13] relies on slowing down the ultrafast label bits by performing serial-to-parallel conversion with optoelectronic devices that support burst-mode operation. We are adopting an in-band 32-bit label modulated at 25 Gbps similar to the packet payload, where the first 16 bits include the destination address, broadcast flag, virtual path identifier, checksum bits, etc., and the LP should recognize them to take a decision on packet forwarding, whereas the other 16 bits include information such as the source address and packet ID and they are to be used at the destination shared buffer to check the received packets. As illustrated in Fig. 2 , the complete label processor (LP) subsystem comprises a set of label extractors (LEs) and a shared controller, where each input port of the optical switch is equipped with an attached LE. The LE extracts a copy of the packet label from the packet incoming at that port, and interfaces it to the shared controller by generating slow parallel signals that are directly supplied to the FPGA-based controller. The incoming packet is thus divided into two parts; one of them goes through the label processor where label recognition takes place and the corresponding control signals for the optical switch are generated, whereas the other part is launched into the optical switch after going through a fiber-based delay to compensate for the label processing time. At the LE, serial-to-parallel conversion (SPC) is performed for all the label bits on a bit-by-bit basis; to enable inputting them to the slow controller. After undergoing serial-to-parallel conversion, the label bits are simultaneously latched into the shared controller where they are checked against the forwarding table stored there. Upon matching a specific table entry, activation is done for the signals that control the desired switch output port so as to get it reconfigured.

 

Fig. 2 The new label processor subsystem.

Download Full Size | PDF

An integrated optoelectronic device called the optically clocked transistor array (OCTA) [13] is utilized for performing the label’s SPC process, where an electrical copy of the optical packet is fed into OCTA and a set of optical triggers is applied to let OCTA perform a bit-by-bit conversion. A principal optical trigger is generated by the optical-trigger pulse generator (TPG) and split into copies with the same number as the label bits. Each copy is then provided with an incremental time delay to match a given bit timing. The delay added to the optical trigger used to convert the n ^th bit of the label can be given as (n-1) × τ, where τ is the time difference between two successive label bits. All the optical triggers are supplied to OCTA via a specially designed optical head. A highly-sensitive burst-mode APD-TIA [14] is used to generate an electrical copy of the incoming packet which is then coupled into a transmission line (TL) in OCTA. Despite that the whole packet is supplied to OCTA, the SPC process is done only for the label part. The bits converted by OCTA are output from different conversion channels and input to separate off-chip comparators. A slow PD present at each LE is used to generate an electrical signal that takes the form of the packet envelop, and by using this signal all the comparators’ outputs are latched simultaneously into the FPGA. Moreover this signal declares the occupancy of a specific switch output-port throughout the whole packet duration, and thus it is essential for detecting contention.

The total power consumed by the LP can be separated into the LEs and controller power. Reducing the controller power is beyond the scope of this work, whereas the more compelling task is the reduction of the LE power, or in particular reducing the power of OCTA and TPG. OCTA has been originally designed to maintain low power consumption during idle times by employing high-value resistances in its integrated circuits so as to limit the power dissipated by idle electrical current. Thus the challenge that should be addressed is to minimize OCTA’s necessary trigger energy on one hand, and to generate optical trigger signals with sufficient energies, and with reduced power consumption and latency of the TPG on the other hand. Several steps were taken to reach this goal, where they can be classified into the following key points; a) devising a novel optical TPG, b) enhancing OCTA’s operation mechanism, and c) supplying OCTA’s optical triggers via back illumination.

3. The key enablers of the new label extractor (LE)

a. Novel optical trigger pulse generator (TPG)

Most of the available burst-mode solutions demand long preamble bits to recover the clock signal of transmitted packet [15], where the output from the clock-data recovery (CDR) unit is used in turn for driving a serializer-deserializer (SerDes). The preamble bits add an overhead that reduces the efficiency of utilizing the available transmission bandwidth. Differently our LP is capable of operating in a real burst-mode manner without demanding any preamble bits. To make the LP satisfy this important requirement, OCTA is designed to operate immediately when an optical trigger is applied, whereas the triggers themselves should be generated without delay upon receiving an incoming packet. The previous way for doing that was to employ copies of the packet-label itself as optical triggers. Separating the label from incoming packet was done by using an electroabsorption modulator (EAM) and other high-speed electronic devices as illustrated in Fig. 3(a) . Upon packet reception, the slow PD output turns high, and a corresponding electrical pulse with a fast rising edge is generated by the following comparator, whereas as the comparators’ two outputs are delayed relative to each other before being input to the AND gate; the AND gate output takes the form of a rectangular pulse with a duration adjusted to match the label duration. The resulting electrical pulse is amplified and used to drive the EAM so as to separate the packet label from the payload. An Erbium-doped fiber amplifier (EDFA), was also used to provide amplification for the separated label after encountering noticeable loss at the EAM e.g., 8 dB, and before getting divided among OCTA’s channels. Using an EDFA together with high-speed electronic devices demanded a continuous supply of electric current even during packet absence; which resulted in a high-power consumption that was almost wasted. Here, as shown in Fig. 3(b) we present a novel TPG in which the EDFA is replaced by a semiconductor optical amplifier (SOA), whereas the EAM and other related devices are all eliminated.

 

Fig. 3 The optical-trigger generation mechanisms; old (a) versus new (b).

Download Full Size | PDF

In addition to the essential demand for reducing the power consumption of the TPG, it is also advantageous –as explained later on– to generate an optical trigger that takes the form of a single optical pulse; not a train of pulses. And what makes this more challenging is the need to have this pulse synchronized to the incoming packet. To fulfill these conditions, the new TPG is designed to provide selective amplification for the first bit of the incoming packet by using an SOA as shown in Fig. 3(b). Unlike the EDFA, the SOA is characterized by fast carrier dynamics that enables it to be turned on and off very quickly and for very short time intervals. The new TPG comprises a custom-made current driver realized as an optoelectronic integrated circuit (OEIC). The OEIC enables the SOA gain for a very short period by allowing electrical current to flow only during that period.

The SOA driver circuit has been developed to generate a narrow pulse (~1ns) of high peak current (>600 mA) for driving the SOA with low power consumption. As shown in Fig. 4 the structure of the SOA driver can be divided into 3 sections; a discharge-based (DB) circuit, an intermediate section for pulse shaping, and then an output section that has the capability of allowing or preventing the flow of high electrical current. The driver OEIC is implemented on indium phosphide substrate and operates by applying a copy of the optical packet as a trigger. A single electrical pulse corresponding to the applied packet is generated, and then undergoes reshaping i.e., pulse broadening and slight amplification, before being finally used to shortly turn on a set of integrated transistors that allows the flow of high electrical current for a precisely controlled time duration.

 

Fig. 4 Illustration for the structure of the SOA-driver OECI.

Download Full Size | PDF

The DB circuit [16] employs an integrated metal-semiconductor-metal (MSM)-PD. When an optical trigger pulse is applied to the MSM-PD, free carriers are generated in it, and as long as a sufficient voltage difference is present between its terminals, the carriers support the flow of current. In the DB circuit, by having the input capacitor C_in initially charged, a high voltage difference is kept between the MSM-PD terminals, whereas when the optical trigger is applied, an electrical current flows causing the discharge of C_in with a corresponding reduction in the voltage difference between the MSM-PD terminals. The current then ceases to flow due to the insufficient voltage difference, whereas an output electrical pulse is formed at the MSM-PD terminal equipped with the resistors R_bias. In the absence of an optical trigger, the MSM-PD turns to be depleted of carriers and its terminals become isolated again. This allows C_in to get slowly charged by V_in through the high resistance R_in. The rise-up time of the generated pulse is very short due to the ultralow capacitance of the MSM-PD. Moreover, if compared to a PIN-PD, the MSM-PD exhibits the advantages of easier fabrication due to its surface structure, and the absence of output when no bias is applied between its terminals. The DB circuit in the SOA driver OEIC is designed to operate with a split from the incoming packet as an optical trigger, and not with a single optical pulse. Upon triggering, a short electrical pulse is formed at the resistors R_bias by the mechanism explained above, whereas no charge will accumulate at the capacitor C_in as long as the packet is kept supplied to the MSM-PD. The DB circuit will produce a single electrical pulse irrespective of the packet pattern i.e., combinations of ‘0’ and ‘1’ bits. During the inter-packet gap, the circuit isolation caused by the MSM-PD’s depletion of carriers allows C_in to get prepared for the next incoming packet by recharging.

High electron mobility transistors (HEMTs) are used in the implementation of the other driver’s sections to enable their fast operation. One SOA terminal is connected to the driver, whereas the other terminal is supplied with a high DC voltage. As the driver’s output section, illustrated here as transistor T_r, controls the flow of current through the SOA, time-selective amplification is provided only when the driver allows current. The driving speed of the SOA is limited by an RL time constant, where L is the inductance present between the driver and SOA. It is worth mentioning that to overcome spending long time for building up a high-level current, electrical circuits are conventionally provided with a continuous supply of electrical current. Such that when the circuit output is off, the current is just thrown away without use, whereas when the output turns on, the current is employed. Thus to cope with the high-speed demand, the current is made available all the time and then undergoes switching, which comes on the expense of high power dissipation. Differently, the new driver operates without supply of high electrical current, as the current driven into the SOA is made of the charge already present at the capacitor C_s, whereas after C_s is discharged a very low current recharges it from the power supply V_high. This very low current is sufficient as the time separating the discharging instants of C_s is long enough e.g., 120 ns. Realizing a fast rise-up time and high current level is essential for the TPG operation mechanism, and to enable that, the pool of electrical charges kept at C_s is place in very close proximity to the SOA.

Figure 5(a) shows the packet arriving at the SOA, whereas the red curve in Fig. 5(b) shows the normalized waveform of the amplified spontaneous emission (ASE) output from the SOA when driven by the current driver while not having an optical input. The resulting waveform almost corresponds to the SOA’s gain profile. Using the SOA for selectively amplifying the first packet bit can be explained as follow. A split from the incoming packet (Fig. 5(a)) is used to trigger the SOA-driver OEIC which in turn enables the SOA to exhibit optical gain by allowing a current pulse. By using a fiber delay line, the arrival of the packet at the SOA can be roughly adjusted to take place around the optical gain peak as shown in Fig. 5. The packet’s first optical pulse will experience strong amplification, and as the SOA’s accumulated carriers are quenched in this stimulated amplification, the following pulses will fade out due to lack of gain as shown in Fig. 5(b). Accordingly the TPG will generate optical trigger pulses such that the first pulse possesses sufficiently higher energy than all the other pulses. It is also evident that the power dissipated by the TPG is limited to a short time interval only when a packet is received and not otherwise. The TPG’s power consumption has been reduced significantly from 3 W previously to 130 mW. Moreover after replacing the EDFA (~10 m) with the shorter SOA, the TPG latency is reduced by several tens of nanoseconds, where compared to 105-ns latency in earlier HOPR prototype, the new LP’s latency is as low as 50 ns.

 

Fig. 5 The SOA’s input (a) and output (b) waveforms.

Download Full Size | PDF

Figure 6 shows the relation between the incoming packet power and the energy of the first resulting bit after amplification, where the SOA’s operation can be identified into the linear and saturation regions. The increase in output energy sharply declines at the saturation region providing the gain-quenching condition that is necessary for having the first amplified pulse sufficiently higher than all the following pulses. Operating in the saturation region is also very important to provide tolerance for fluctuation in input packet power, where the corresponding change in the energy of the resulting optical trigger pulse is limited. As detailed later on, the operation of the full label processor subsystem has been demonstrated with a 5.6-pJ optical trigger pulse generated by the new TPG. This value corresponds to 0.35 pJ per conversion channel, or in other words 0.35 pJ for the conversion of each bit in the packet label.

 

Fig. 6 The energy of TPG’s first output pulse versus the input packet power.

Download Full Size | PDF

b. Optically clocked transistor array (OCTA)

The structure of OCTA is illustrated in Fig. 7 , in which 16 SPC channels are connected to a shared transmission line (TL); each via a separate HEMT (T_m). The gate terminal (G) of the transistor T_m is controlled by a DB-MSMPD trigger circuit similar to the one employed at the SOA-driver circuit but with different design parameters. OCTA’s proper operation demands T_m to be at the on-state for a very short time interval so as to convert only a single bit, and thus the DB-MSMPD trigger circuit should produce a single narrow electrical pulse. The packet-label has been previously used as the optical trigger, and to enable operation under this condition, the charge recovery time of C_in is made sufficiently longer than label length i.e., 640 ps, by setting R_in high, and more importantly the energy of the first label bit was made high enough e.g., 2 pJ, so as to completely discharge C_in.

 

Fig. 7 The structure of OCTA and illustration of its operation schemes.

Download Full Size | PDF

As shown in Fig. 7(a), OCTA’s SPC was initially performed by the sample-and-hold (S&H) scheme [13], where an electric charge corresponding to the considered bit level is sampled into the capacitor C_hold through T_m. The voltage-change induced at C_hold is then amplified by a high-gain inverter amplifier to produce the channel’s final output. This SPC scheme suffers from the limited difference between the sampled charge that corresponds to the ‘1’ and ‘0’ bits, respectively. The reason is that charging C_hold i.e., in case of a ‘1’ bit, increases the voltage at the source terminal of T_m and thus forces it to get turned-off. Moreover, charging C_hold cannot be done efficiently as a part of the bit voltage is unavoidably dissipated in the TL’s characteristic impedance that is always present in parallel to the turned-on conversion channel.

The S&H scheme was then replaced by the discharge-or-hold (DoH) scheme [17]. Figure 7(b) illustrates the operation of the DoH scheme, where the charge initially present at C_hold is either discharged into the TL or kept unchanged. Discharging C_hold is done more efficiently than charging it, and a much higher difference of charge is produced at C_hold. In this work, the benefit of the DoH scheme has been further elevated by using a label signal with negative voltage span i.e., negative voltage for a ‘1’ bit and zero voltage for a ‘0’ bit. Figure 8 illustrates a comparison for performing serial-to-parallel conversion with the DoH scheme while using a TL signal either with a positive or negative polarity. The gate voltage signal generated by the optical trigger pulse is highlighted in blue. When the transistor T_m is turned on i.e., V_GS exceeds the threshold voltage V_th, the voltage difference between its drain and source terminals i.e., ΔV_DS, is obviously higher in case of using a TL signal with negative polarity. This allows the capacitor C_hold to get discharged more efficiently because of the higher electrical current enabled by the higher ΔV_DS. Then if the energy of optical trigger pulse is reduced, a corresponding reduction in the gate pulse amplitude takes place as highlighted by the dotted line. Even with this reduction in ΔV_GS, the initially higher value of ΔV_DS enables the conversion to be as efficient as the case of using a positive polarity signal with unreduced optical trigger energy. Following this modified method, a highly sensitive OCTA device that can operate with sub-picojoule optical trigger energy has been achieved.

 

Fig. 8 Carrying out the DoH scheme with TL signals of positive (a), and negative polarities.

Download Full Size | PDF

The effect of optical trigger’s shape on OCTA’s operation is illustrated in Fig. 9 , where the optical trigger waveforms are shown in red, and the resulting electrical waveforms are shown in blue. The threshold voltage V_th of the transistor T_m is given as a dotted line, and only the electrical pulse that sufficiently exceeds this line can properly turn on T_m to enable conversion. When the optical trigger pulses have the same amplitude i.e., the case of using a copy of the packet-label as a trigger, only a high pulse-energy can produce a single electrical pulse at the gate of T_m, whereas due to the incomplete discharging of C_in with lower optical pulse energies, a non-useful waveform similar to Fig. 9(c) would be generated. In this case, a negative DC voltage bias can be added to eliminate the effect of the second electrical pulse by keeping it under V_th, but as the difference between the first and second electrical pulses is low, the remaining part of the first pulse above V_th won’t properly turn T_m on. Differently, when the first optical trigger pulse is sufficiently higher than all the following pulses i.e., the new TPG case, low pulse energies can still result in a useful waveform similar to Fig. 9(e).

 

Fig. 9 The MSM-PD’s optical triggers (a, d), and corresponding electrical signals (b, c, e), in case of the MSM-PD’s front illumination, and the dotted line represents V_th of transistor T_m.

Download Full Size | PDF

c. OCTA’s back illumination

Figure 10(a) shows a photo for the old packaged OCTA, where an optical head was used to supply the optical triggers from the chip’s top side, and as illustrated the optical head includes an array of lenses to focus the optical triggers on the MSM-PDs located at the chip front side. This way for supplying the optical triggers to the chip has several shortcomings such as the need for active alignment, the difficulty of assembling the used optical head, and more importantly the problem of wasting a large portion of the applied optical trigger energy due to the shadowing effect that occurs for front-illuminated light by the MSM-PD’s interdigitated metal electrodes [18]. Thus, here we are introducing a new packaging method to overcome the aforementioned issues. Figure 10(b) shows a photo for OCTA’s new package mounted on the LP electrical board, where the packaged chip is attached to the optical head from above.

 

Fig. 10 The LP PCB and illustration of optical supply method; old (a) versus new (b).

Download Full Size | PDF

Before packaging, the thickness of the used chip was first reduced to ~130 μm by lapping its substrate, and after supplying the optical trigger via the chip’s backside, the responsivity of the MSM-PD’s has been almost doubled; enabling the trigger pulse energy to be cut by half. The used optical head is a commercially available fiber-array block with a pitch of 250 μm, where the trigger-transferring fibers are positioned at the same spacing as the chip’s MSM-PDs. Unlike the case of having a fiber-air interface, the higher refractive index of OCTA’s InP substrate prevents the quick divergence of the optical beam launched from single-mode fibers (SMFs), and a spot size that matches the MSM-PD’s effective area is achievable with a thin substrate. For a substrate thickness of ~130 μm, the 1/e² diameter of the resulting optical spot is ~13 μm, whereas the active area of the MSM-PD has a diameter of 20 μm. Thus even without using lenses, the resulting optical spot size is sufficiently narrow to fit into the active area while providing large misalignment tolerance.

Previously, when front illumination was used as shown in Fig. 10(a), the alignment of triggers to chip could only be done by using an active method, where the device was turned on and the position of optical head was adjusted by observing the 16 device outputs. Thus to find the best alignment position, a cumbersome process of optimizing the output from all 16 channels had to be done. However, by currently using back illumination instead, no opaque object is placed anymore above the chip and thus a much easier passive alignment method can be realized. When an optical signal is applied to the MSM-PD via its backside, a part of this signal goes though the MSM-PD and can still be seen from the front side of the chip. The new alignment process is simply done by simultaneously observing the optical signals going through the MSM-PDs of all 16 conversion channels using an infrared camera focused on the chip surface. The best alignment position is simply found when 16 clear optical spots are visually observed. Using the new optical head has also significantly contributed to cost reduction; as the assembling time required for the new head is much shorter than the previous one where the alignment of lens arrays to other head components is not as simple as placing fibers in grooves. Figure 11(a) shows the direct output of the packaged chip for ‘1’ and ‘0’ bits with optical trigger energy of 0.35 pJ/bit. By adjusting the threshold voltage of the comparator attached to this conversion channel, the comparator’s outputs shown in Fig. 11(b) were obtained and correct recognition of the ‘1’ and ‘0’ bit levels has been achieved.

 

Fig. 11 (a) OCTA’s measured direct output, and (b) comparators’ output.

Download Full Size | PDF

4. Conclusion

A label processor subsystem for 100-Gbps (25-Gbps × 4λs) burst-mode optical packets has been developed based on an enhanced optoelectronic serial-to-parallel converter and a novel optical-trigger-pulse generator (TPG). Significant reduction in the total power consumption and latency of the whole subsystem has been achieved via three key achievements; 1) based on a novel operation mechanism for amplification with bit-level selectivity, an optical trigger pulse generator comprising an SOA and a custom-made current driver OEIC, is realized such that it consumes power for a very short duration only upon packet arrival, 2) the energy of optical triggers needed by the optoelectronic serial-to-parallel converter is reduced by utilizing a negative-polarity signal together with employing an enhanced serial-to-parallel conversion scheme entitled the discharge-or-hold scheme, 3) the necessary optical trigger pulse energy is further cut down by half by coupling the triggers via the chip’s backside, whereas a novel lens-free packaging method is developed to enable a low-cost alignment process that works with simple visual observation. Under the new conditions, each channel in the optoelectronic converter has been efficiently operated with an optical trigger pulse energy of 0.35 pJ/bit compared to a previously necessary value of 2.0 pJ/bit, whereas the TPG’s novel operation scheme reduced its power consumption from previously 3W down to 130mW, and enabled a total latency of as low as 50 ns. We believe that the new label processor subsystem is a keystone for realizing a large scale HOPR-based DC network.