## Abstract

We experimentally investigate the soliton formation and dynamics in the nonlinear propagation of the generated signal and probe beams in four-wave mixing (FWM) process with atomic coherence in a three-level atomic system, under the competition between focusing and defocusing nonlinearities, as well as between gain and dissipation, due to the third- and fifth-order nonlinear susceptibilities with opposite signs. With multi-parameter controllability and nonlinear competition in the system, fundamental, dipole, and azimuthally-modulated vortex FWM solitons can transform mutually from one to the other. Such investigations have potential applications in optical pattern formation and control, and all-optical communication.

© 2015 Optical Society of America

## 1. Introduction

The response of a central symmetric medium to incident light fields can often be expanded as $P={\Sigma}_{m\ge 0}{\chi}^{(2m+1)}{E}^{(2m+1)}$, in which ${E}^{(2m+1)}$ ($m=1,2,\mathrm{...}$) can be a mixing product of various light fields [1]. Typically, only the linear ($m=0$) and third-order ($m=1$) terms are considered, since the magnitude of ${\chi}^{(2m+1)}$ decreases rapidly with larger *m*. During the past decade, many techniques were exploited to enhance higher-order nonlinearities, such as by making use of the effect of electromagnetically induced transparency (EIT) [2]. As a fascinating phenomenon in nonlinear systems, optical solitons have been investigated intensively, and different types of solitons with different forms of nonlinearities have been demonstrated [3–8]. It is worth noting that the multi-dimensional spatial solitons are not stable just with third-order nonlinearity because of the catastrophic self-focusing effect [3,4]. In order to avoid this undesirable effect, several types of nonlinearities [9,10] or elliptic beam with orbital angular momentum [11] were proposed to overcome such instability. One of the schemes is to consider the focusing third-order and defocusing fifth-order nonlinear effects simultaneously with relatively high beam intensity, i.e., employ the competing cubic-quintic (CQ) nonlinearities [12], which was experimentally reported in photoconductivity and degenerate four-wave-mixing (FWM) measurements [13,14], *π*-conjugated polymer [15], chalcogenide glasses [16], and organic materials [17]. Unfortunately, such nonlinearity is always accompanied by significant higher-order multiphoton processes [6] that will affect the beam propagation. In [19,20], the multi-dimensional solitons and light condensates were theoretically studied in atomic system with giant CQ nonlinearity enhanced by EIT. Nevertheless, no further experimental investigation was reported so far.

In this letter, we first experimentally demonstrate controllable soliton formation and dynamics of the FWM signal and probe beams with competing CQ nonlinearities, in which the competitions between nonlinear cubic gain and linear as well as quintic absorptions are also involved. Comparing with pervious literatures, the main novelties of this letter can be summarized as the following two points: (I) The CQ nonlinear competition not only shows up in the real parts (refraction) but also in imaginary parts (absorption/gain). What’s more, the competitive coefficients can be controlled with multiple parameters (e.g., the frequency detunings, powers of pump fields, etc.) in the chosen atomic medium. (II) The EIT-enhanced competitions are investigated in stabilizing fundamental soliton, and high-order dipole and azimuthally-modulated vortex (AMV) soliton of the FWM beam by controlling experiment configuration. Especially for the AMV case with the configuration being fixed, it is found that the CQ competition mechanism also can cause the transformation among fundamental, dipole and AMV solitons, which can be stabilized by using the EIT-enhanced fifth-order nonlinearity with relatively low intensity.

The paper is organized as follows. In section 2, we present the theoretical model and experiment scheme. In Section 3, the competition between CQ nonlinear in soliton formation and dynamics of the FWM signal and probe beams are investigated. In Section 4, we conclude the paper.

## 2. Basic theory and experimental scheme

The experiments are carried out in a Na atomic vapor oven. As shown in Fig. 1(a), a three-level atomic system is formed by energy levels $|0\u3009$ ($3{S}_{1/2}$), $|1\u3009$ ($3{P}_{3/2}$) and $|2\u3009$ ($4{D}_{3/2}$) and five laser beams are adopted. The laser beams ${E}_{1}$, ${{E}^{\prime}}_{1}$ and ${E}_{p}$ (${E}_{2}$ and ${{E}^{\prime}}_{2}$) connect the transition between levels $|0\u3009$ and $|1\u3009$ ($|2\u3009$) with the atomic resonant frequency ${\Omega}_{1}$ (${\Omega}_{2}$). The setup is shown in Fig. 1(b), and the configuration of the laser beams is given by the Fig. 1(c). The pump beams ${E}_{1}$ and ${{E}^{\prime}}_{1}$ with wavevectors ${k}_{1}$ and ${{k}^{\prime}}_{1}$ (with nearly the same power ${P}_{1}$, frequency ${\omega}_{1}$) propagate in the opposite direction to the weak probe beam ${E}_{p}$ (${k}_{p}$,${P}_{3}$, frequency ${\omega}_{3}$) with small angles (${\theta}_{1}={0.3}^{o}$) between them. The three beams come from the same dye laser DL1 (10 Hz repetition rate, 5 ns pulse width, and $0.04c{m}^{-1}$ linewidth) with frequency detuning ${\Delta}_{1}={\omega}_{1}-{\Omega}_{1}$, and their wave-vectors are in the *x-o-z* plane. The other two pump beams ${E}_{2}$ and ${E}_{2}\text{'}$ (${k}_{2}$ and ${{k}^{\prime}}_{2}$, ${P}_{2}$ and ${{P}^{\prime}}_{2}$, ${\omega}_{2}$) from another dye laser DL2 (which has the same characteristics as the DL1) with ${\Delta}_{2}={\omega}_{2}-{\Omega}_{2}$ also propagate along the opposite direction of ${E}_{p}$ with small angles (${\theta}_{2}={0.3}^{o}$) between them, and their wave-vectors are in the *y-o-z* plane. In this system, there will be one FWM process, that the one-photon resonant degenerate FWM process (Fig. 1(a)) satisfies the phase-matching condition ${k}_{F}={k}_{p}+{k}_{1}-{{k}^{\prime}}_{1}$ with the generated signal field ${E}_{F}$ propagating nearly in the opposite direction to the beam ${{E}^{\prime}}_{1}$. In the experiments, ${E}_{p}$ and ${E}_{F}$ are split by beam-splitter mirrors and recorded by charge coupled device (CCD) cameras and photo-multiply tubes (PMTs) simultaneously, as shown in Fig. 1(b).

In such experimental arrangement, several input laser beams with spatially non-uniform intensity and generated FWM signals interact with each other in a large spatial region, therefore spatially-varying phase modulation can affect the propagation and spatial patterns of the probe and FWM signals, which are relatively weak compared to the pump beams. The propagations of the probe and FWM signals can be described by the set of coupled equations [8]:

*p*and

*F*are corresponding to the probe and ${E}_{F}$ signals, respectively. The second term in the right-hand side of Eq. (1a) and Eq. (1b) describes the conversion between the probe and FWM signals. In Eq. (1a)-(2b), ${x}_{p,F}^{(1)}$ represents the linear susceptibility of the probe or FWM signals, ${\chi}_{p,F}^{(n){S}_{p,F}}$ ($n=3,5$) the

*n*th-order self-phase modulation (SPM) nonlinear susceptibility, and ${\chi}_{p,F}^{(n){X}_{j}}$ the

*n*th-order cross-phase modulation (XPM) nonlinear susceptibility excited by the pump field ${E}_{j}$ (${{E}^{\prime}}_{j}$). All these susceptibilities can be obtained by solving the density-matrix equations for the three-level atomic medium [22]. Here, we just show three typical expressions, which are involved linear, third-order, and fifth-order nonlinear susceptibilities in Eq. (1a) and Eq. (1b) as following:

*N*is the atomic density, ${\Gamma}_{ij}$ denotes the population decay rate between the corresponding energy levels $|i\u3009$ and $|j\u3009$, and ${\mu}_{10}$ is the electric dipole moment. ${E}_{j}$ (${{E}^{\prime}}_{j}$) is the electrical field amplitude of the corresponding field, which is spatial dependence as ${E}_{j}({r}_{\perp})$ (${{E}^{\prime}}_{j}({r}_{\perp})$), with ${r}_{\perp}$ being the transverse dimensions perpendicular to the propagation direction $z$. Combining Eqs. (3a)-(3c), the total susceptibility can be written as ${\chi}_{p,F}={\chi}_{p,F}^{(1)}+{\chi}_{p,F}^{(3)}{\left|E\right|}^{2}+{\chi}_{p,F}^{(5)}{\left|E\right|}^{4}$.

In Fig. 2(a), we present the theoretical nonlinear refractive index of the system versus ${\Delta}_{2}$ and the total pump intensity. In Figs. 2(b) and (c), the dependences of the real and imaginary parts of the linear term ${\chi}_{p,F}^{(1)}$, third-order one ${\chi}_{p,F}^{(3){X}_{2}}\left({\left|{E}_{2}\right|}^{2}+{\left|{{E}^{\prime}}_{2}\right|}^{2}\right)$ and fifth-order one ${\chi}_{p,F}^{(5){X}_{2}}\left({\left|{E}_{2}\right|}^{4}+{\left|{{E}^{\prime}}_{2}\right|}^{4}+{\left|{E}_{2}\right|}^{2}{\left|{{E}^{\prime}}_{2}\right|}^{2}\right)$ on the total pump intensity $\left({\left|{E}_{2}\right|}^{2}+{\left|{{E}^{\prime}}_{2}\right|}^{2}\right)$ are shown. It is obvious that ${\chi}_{p,F}^{(1)}$ keeps constant with varying ${G}_{2}$ (${{G}^{\prime}}_{2}$). Determined by the real parts of the third- and fifth-order susceptibilities, respectively, the focusing and defocusing nonlinearities follow cubic- and quintic-dependent laws, but the former case is positive and the latter case negative. The total refractive index composed of the linear part and the XPM nonlinear parts due to ${E}_{2}$ and ${{E}^{\prime}}_{2}$ can be further obtained as $\Delta n={n}_{1}+{n}_{2}^{{X}_{2}}({I}_{2}+{{I}^{\prime}}_{2})+{n}_{4}^{{X}_{2}}({I}_{2}^{2}+{{I}^{\prime}}^{2}{}_{2})$, where ${n}_{2}^{{X}_{2}}=\mathrm{Re}[{\chi}_{p,F}^{(3){X}_{2}}]/({\u0454}_{0}c{n}_{1})>0$ and ${n}_{4}{}^{{X}_{2}}=\{4\mathrm{Re}[{\chi}_{p,F}^{(5){X}_{2}}]-{\u0454}_{0}^{2}{c}^{2}{({n}_{2}^{{X}_{2}})}^{2}\}/(2{\u0454}_{0}^{2}{c}^{2}{n}_{1})<0$ [22] are the corresponding Kerr nonlinear coefficients, respectively, with ${I}_{2}={\u0454}_{0}c{\left|{E}_{2}\right|}^{2}/2$ (${{I}^{\prime}}_{2}={\u0454}_{0}c{\left|{{E}^{\prime}}_{2}\right|}^{2}/2$) being the intensity determined by the power ${P}_{2}$ (${{P}^{\prime}}_{2}$). So when ${I}_{2}$ (${{I}^{\prime}}_{2}$) is not large enough, the refractive index $\Delta n$ will increase with ${I}_{2}$ (${{I}^{\prime}}_{2}$) due to the dominant positive third-order nonlinearity increasing in cubic law, but as ${I}_{2}$ (${{I}^{\prime}}_{2}$) exceeds a threshold value and gets into high-power region, it turns to decrease because that the negative fifth-order nonlinearity will play a dominant role on $\Delta n$. As shown in Fig. 2(a), this competition between different nonlinearities can be controlled by both the pump field detuning ${\Delta}_{2}$ and ${I}_{2}$ (${{I}^{\prime}}_{2}$). Thus, in the propagation of the probe or FWM beams, the diffraction will accompany competing nonlinearities between focusing and defocusing for certain pump field intensity, which is crucial for yielding high-order stable spatial solitons. The imaginary parts of the susceptibilities that offer gain or loss to the probe or FWM signals also have a competition which can influence the propagation of these beams. As presented in Fig. 2(c), when the pump intensities are small to satisfy $\left|Im\left\{{\chi}_{p,F}^{(1)}\right\}\right|\gg \left|Im\left\{{\chi}_{p,F}^{(3){X}_{2}}{\left|E\right|}^{2}\right\}\right|\gg \left|Im\left\{{\chi}_{p,F}^{(5){X}_{2}}{\left|E\right|}^{4}\right\}\right|$, the linear dissipation will be the dominant one. However, with ${I}_{2}$ (${{I}^{\prime}}_{2}$) increasing, the satisfied conditions of $\left|Im\left\{{\chi}_{p,F}^{(3){X}_{2}}{\left|E\right|}^{2}\right\}\right|\gg \left|Im\left\{{\chi}_{p,F}^{(1)}\right\}\right|$ and $Im\left\{{\chi}_{p,F}^{(3){X}_{2}}{\left|E\right|}^{2}\right\}<0$ can make the third-order nonlinear gain be dominant. If ${I}_{2}$ (${{I}^{\prime}}_{2}$) further increases, $Im\left\{{\chi}_{p,F}^{(5){X}_{2}}{\left|E\right|}^{4}\right\}>0$ will make the fifth-order nonlinear dissipation be dominant. So, competitions exist not only among the real parts but also among the imaginary parts of the third- and fifth-order nonlinearities.

In the experiment, the power of ${{E}^{\prime}}_{1,2}$ is approximately 2 times stronger than that of ${E}_{1,2}$, and 10 times stronger than that of ${E}_{p,F}$, therefore the SPM due to ${E}_{p,F}$ can be neglected. Furthermore, neglecting diffraction, we can obtain a set of solutions of ${E}_{p,F}$, in which the nonlinear phase shifts introduced in the propagation are ${\varphi}_{p,F}=2{k}_{p,F}z{\Sigma}_{m=1,2}{\Sigma}_{j=1,2}[{n}_{2m}^{{X}_{j}}({I}_{j}^{m}+{{I}^{\prime}}_{m}{}^{m})]/({n}_{0}{I}_{p,F})$. The additional transverse propagation wave-vector can be obtained as $\delta {k}_{x}=(\partial {\varphi}_{p,F}/\partial x)\widehat{x}$, where $\widehat{x}$ is the unit vector along the horizontal axes. The direction of $\delta {k}_{x}$ determines the horizontal propagation of the beam, while the phase-front curvature ${\partial}^{2}{\varphi}_{p,F}/\partial {x}^{2}<0$ (${\partial}^{2}{\varphi}_{p,F}/\partial {x}^{2}>0$) lead to local focusing (defocusing) of the beams imposed on the weak field by the strong filed. Under balanced diffraction and XPM, the probe and FWM signals can form solitons with the electrical field intensity profile generally expressed as

*l*the topological charge. If the angles between ${E}_{1}$,(${E}_{2}$)${{E}^{\prime}}_{1}$,(${{E}^{\prime}}_{2}$) and ${E}_{p}$ are all sufficiently small, $\gamma =0$, $l=0$ and fundamental mode solitons of ${E}_{p,F}$ are obtained. If the angle between ${E}_{p}$ and ${E}_{1}$ (${E}_{2}$) is small, but the one between ${E}_{1}$ (${E}_{2}$) and ${{E}^{\prime}}_{1}$ (${{E}^{\prime}}_{2}$) is sufficiently large to introduce the so-called EIG because of interference [8], $\gamma =1$, $l=1$, $K=1$ and dipole mode soliton are then obtained. However, as the angles are all sufficiently large, $\gamma \ge 3/2$, $l=1$, $0\le K\le 1$ and azimuthally-modulated vortex (AMV) solitons appear. Besides by the switch among different beams configuration, such transformations among different types of soliton also can be acquired by competing CQ nonlinearities, which will be shown in the Fig. 4.

## 3. Experimental observation of cubic-quintic solitons

First, the images of the transmission of the probe beam versus ${\Delta}_{2}$, captured by CCD cameras with different beams or combinations turned on, are shown in Figs. 3(a1)-(a4), and the corresponding intensities versus ${\Delta}_{2}$ recorded by PMT are shown in Fig. 3(a5). When one of or both of ${{E}^{\prime}}_{2}$ and ${E}_{2}$ are on, it is obvious that the probe suffers more significant focusing around the two-photon resonant (TPR) point (${\Delta}_{1}+{\Delta}_{2}=0$) than in other regions along the ${\Delta}_{2}$-axis, since the large XPM nonlinearity from ${{E}^{\prime}}_{2}$ and ${E}_{2}$ can bring strong focusing. The total nonlinear refractive index $\Delta {n}^{X2}=\Delta {n}_{2}^{X2}+\Delta {n}_{4}^{X2}$, i.e., the sum of third- and fifth-order nonlinear refractive indices, is positive and very large when ${\Delta}_{2}\u0454[3050]GHz$, which undergoes sharp decrease when ${\Delta}_{2}$ is far away from TPR. This explanation can be further verified by the images with ${{E}^{\prime}}_{2}$ and ${E}_{2}$ both off, which shows no variation of the probe transmission images versus ${\Delta}_{2}$ due to the absence of ${\Delta}_{2}$-dependent Kerr nonlinearity (Fig. 3(a4)). It is interesting that the image focusing is enhanced in Fig. 3(a2) compared with Figs. 3(a1), and the focusing in Fig. 3(a3) is almost the same as that in Fig. 3(a1), though the total pump power in Fig. 3(a1) with ${{E}^{\prime}}_{2}$ and ${E}_{2}$ both turned on is larger than those in Fig. 3(a3) with one of the beams turned on. To explain this abnormal phenomenon, the fifth-order nonlinearity should be considered, i.e., the competition between CQ nonlinearities mentioned above. Specifically, in Fig. 3(a1), with both ${{E}^{\prime}}_{2}$ and ${E}_{2}$ turned on, the third-order nonlinear refractive index $\Delta {n}_{2}^{X2}={n}_{2}^{X2}({I}_{2}^{}+{{I}^{\prime}}_{2}^{})$ can be effectively weakened by the fifth-order one $\Delta {n}_{4}^{X2}={n}_{4}^{X2}({I}_{2}^{2}+{{I}^{\prime}}_{2}^{2})$. Compared to the case in Fig. 3(a1), we find $\Delta {n}_{4}^{X2}$ suffers more reduction than $\Delta {n}_{2}^{X2}$ when ${E}_{2}$ is turned off (Fig. 3(a2)), which weakens the compensative effect of $\Delta {n}_{4}^{X2}$ on $\Delta {n}_{2}^{X2}$, thus $\Delta {n}^{X2}$ increases and results in more significant focusing. In Fig. 3(a3), $\left|\Delta {n}_{2}^{X2}\right|=\left|{n}_{2}^{X2}{I}_{2}\right|$ and $\left|\Delta {n}_{4}^{X2}\right|=\left|{n}_{4}^{X2}{I}_{2}^{2}\right|$ have nearly equal reductions as compared to the ones in Fig. 3(a1), so $\Delta {n}^{X2}$ has nearly no variation and the nonlinear focusing keeps. The three dots on the solid curve in Fig. 2(b) can elucidate the changing of focusing strength in Figs. 3(a1)-(a3). This explanation can be further verified by the more significant defocusing effect around TPR when ${{p}^{\prime}}_{2}$ is further increased to a higher value as shown in Fig. 3(b) from bottom to top. Satisfying the quintic-law, the decreasing factor of $\Delta {n}_{4}^{X2}={n}_{4}^{X2}({I}_{2}^{2}+{{I}^{\prime}}_{2}^{2})$ is larger than that of $\Delta {n}_{2}^{X2}={n}_{2}^{X2}({I}_{2}^{2}+{{I}^{\prime}}_{2})$, so $\Delta {n}^{X2}$ increases, which leads to the more significant focusing as shown in Figs. 3(b1)-(b6).

The intensity curves also reveal anomaly as shown in Fig. 3(a5), in which the EIT peak with ${{E}^{\prime}}_{2}$ or ${E}_{2}$ individually on is higher than that with both ${{E}^{\prime}}_{2}$ and ${E}_{2}$ on, though the total pump fields intensity is lower in the former case than in the latter case. The reason is that according to Eq. (3c) and the fifth-order curve (the dashdot curve) in Fig. 2(c), the fifth-order nonlinear loss $\mathrm{Im}\left\{{\chi}_{p,F}^{(5){X}_{2}}{\left|E\right|}^{4}\right\}$ is larger with both ${{E}^{\prime}}_{2}$ and ${E}_{2}$ on than with ${{E}^{\prime}}_{2}$ or ${E}_{2}$ individually on, which can effectively lower the transmission peak. When both ${{E}^{\prime}}_{2}$ and ${E}_{2}$ are off, there is only the linear susceptibility and the nonlinear absorption nearly is eliminated, so the EIT peak disappears.

Also, the spatial shift of probe beam due to the cross nonlinear modulation of ${{E}^{\prime}}_{2}$ and ${E}_{2}$ can be observed. In Fig. 3(a1), when ${{E}^{\prime}}_{2}$ and ${E}_{2}$ are both on, the nonlinear phase shift introduced by the two fields is ${\varphi}_{p}=2\Delta {n}^{X2}{k}_{p}z{e}^{-{r}^{2}/2}/({n}_{0}{I}_{p})$ and the transverse nonlinear wave-vector ${\delta}_{r}=-{\nabla}_{r}{\varphi}_{p}$ is the gradient of nonlinear phase shift [23,24]. Because the probe beam spot is within the wings of ${{E}^{\prime}}_{2}$ and ${E}_{2}$ along the *y*-axis, the ${\delta}_{r}$ can be approximated by ${\delta}_{r}=2\Delta {n}^{X2}{k}_{p}rz{e}^{-{r}^{2}/2}/({n}_{0}{I}_{p})$ which leads to the upward shift of probe spots, due to positive $\Delta {n}^{X2}$. When ${{E}^{\prime}}_{2}$ is turned off (Fig. 3(a2)), similar to the nonlinear focusing mentioned above, $\Delta {n}^{X2}$ is larger than that in Fig. 3(a1), so the upward shift displacement becomes larger. With ${E}_{2}$ off in Fig. 3(a3), $\Delta {n}^{X2}$ is almost the same as that in Fig. 3(a1), but smaller than that in Fig. 3(a2), so the shift displacement is similar to that in Fig. 3(a1). When ${{E}^{\prime}}_{2}$ and ${E}_{2}$ are both off, ${\delta}_{y}=0$ for the XPM effect supplied by strong sufficiently pump fields diminishes and the spatial shift disappears (Fig. 3(a4)). Also, in the decreasing of ${{P}^{\prime}}_{2}$ from Figs. 3(b1)-(b6), the spatial shift displacement gradually increases due to the increasing $\Delta {n}^{X2}$.

Next, we adjust the beam configuration to make the FWM signals form dipole solitons and investigate their dynamics under competing third- and fifth-order nonlinearities. When ${\Delta}_{2}$ is scanned in the region far away from TPR, the nonlinearity induced by ${{E}^{\prime}}_{2}$ and ${E}_{2}$ is very weak, so the dipole modulation effect of electromagnetically induced grating (EIG) induced by ${{E}^{\prime}}_{1}$ and ${E}_{1}$ is dominant and ${E}_{F}$ gets dipole pattern [8]. This is verified by the almost unchanged dipole soliton patterns of ${E}_{F}$ when ${\Delta}_{2}$ is within the regions of $[-8010]GHz$ and $[70160]GHz$ in Figs. 3(c1)-(c4). When ${\Delta}_{2}$ is set within the region of $[1070]GHz$, i.e., around TPR, the nonlinear refractive index induced by ${{E}^{\prime}}_{2}$ and ${E}_{2}$ on the FWM signal is positive, which causes the dipole mode of ${E}_{F}$ to collapse into a single spot due to the strong focusing effect. When ${{E}^{\prime}}_{2}$ and ${E}_{2}$ are blocked (Fig. 3(c4)), ${E}_{F}$ remains to be in dipole mode even around TPR due to the vanishing nonlinear focusing effect. Compared to the nonlinear focusing in Fig. 3(c1), the focusing in Fig. 3(c2) with ${E}_{2}$ blocked is stronger and that in Fig. 3(c3) with ${{E}^{\prime}}_{2}$ blocked weaker. The reason is also that the total refractive index with ${{E}^{\prime}}_{2}$ off is smaller than that with ${E}_{2}$ off, but larger than that with both beams on. Due to the enhancement effect of the pump fields, the intensity of the FWM signal shows a peak higher than the background when ${{E}^{\prime}}_{2}$ and ${E}_{2}$ are blocked. With ${{E}^{\prime}}_{2}$ off, the enhancement effect is weakened because of the decreasing of the pump field intensity, so the peak becomes lower; when ${E}_{2}$ is off, the enhancement effect is weaker than that with ${{E}^{\prime}}_{2}$ and ${E}_{2}$ both on, but stronger than that with ${{E}^{\prime}}_{2}$ off, however the peak keeps almost the same height with that when ${{E}^{\prime}}_{2}$ is off. The main reason is that the magnitude of fifth-order nonlinear loss with ${E}_{2}$ off is much larger than that with ${{E}^{\prime}}_{2}$ off, which can bring considerable nonlinear loss and effectively limit the rising of peak. This variation reveals that the competition between the enhancement and nonlinear absorption plays an important role on the final received FWM signal.

With ${{E}^{\prime}}_{2}$ and ${E}_{2}$ both on, we investigate the images of the FWM signal with an increasing power ${{P}^{\prime}}_{2}$. The experimental results are summarized in Fig. 3(d). It is obvious that generated FWM beam is characterized by the dipole mode and single-spot at far away from TPR and around the resonance remain, respectively. But we find that the single-spot focuses more significantly with ${{P}^{\prime}}_{2}$ increasing (Fig. 3(d)), though the magnitude of $\Delta {n}_{4}^{X2}$ increases more rapidly than $\Delta {n}_{2}^{X2}$, which will decrease the spot focusing. The reason for this contradiction is that the imaginary parts of ${\chi}^{(3)}$ and ${\chi}^{(1),(5)}$ are negative and positive, respectively. The nonlinearity of ${{E}^{\prime}}_{2}$ is strong around TPR, and the positive imaginary part of susceptibilities will lead to a considerable dissipation of the FWM signal, which will be easily affected by the nonlinearity according to ${\varphi}_{F}=2\Delta {n}^{X2}{k}_{F}z{e}^{-{r}^{2}/2}/({n}_{0}{I}_{F})$. In Fig. 3(d), the total imaginary part of the linear and nonlinear responses is positive and increases with increasing ${{P}^{\prime}}_{2}$ because the initial ${{P}^{\prime}}_{2}$ is set to achieve $\left|Im[{\chi}_{F}^{(5)}{\left|{E}_{2}\right|}^{4}]\right|>\left|Im[{\chi}_{F}^{(3)}{\left|{E}_{2}\right|}^{2}]\right|$, which will lead to the decrease of the FWM signal intensity. Thus, ${I}_{F}$ decreases due to increasing loss with increasing ${{P}^{\prime}}_{2}$, and the decreasing rate is larger than the increasing rate of $\Delta {n}^{X2}$, which leads to a larger ${\varphi}_{F}$ that the FWM signal around TPR exhibit more significant nonlinear focusing as shown in Figs. 3(d1)-(d7). The iso-surfaces shown in Figs. 3(e) and (f) display the fundamental soliton changing with blockage of different beams and the transformation between dipole and fundamental solitons versus ${\Delta}_{2}$ based on Eq. (4) due to the competitions of nonlinearities and gain/loss mechanisms.

The influence of nonlinear competition on the transformation among fundamental, dipole and AMV solitons is also investigated. If we adjust the spatial configuration of the beams, the AMV mode of the ${E}_{F}$ signal can be obtained. The formation such AMV mode is due to the spatial modulation of the interference patterns [25] among ${E}_{1}$, ${{E}^{\prime}}_{1}$ and ${E}_{p}$. The AMV FWM field can be rewritten from Eq. (4) as

*K*in Eq. (4) is replaced by $\beta $ in Eq. (5), and $\gamma $ is replaced by ${\gamma}_{m}$ which is also determined by the nonlinearities due to ${{E}^{\prime}}_{2}$ and ${E}_{2}$. Specifically, the coefficients $\alpha $, $\beta $, and ${\gamma}_{m}$ can be expressed as $\alpha =({n}_{2}^{X2}{I}_{2}+{n}_{4}^{X2}{I}_{2}^{2})+\left|({\Delta}_{1}+{\Delta}_{2}){\Delta}_{1}\right|({n}_{2}^{X2}{{I}^{\prime}}_{2}+{n}_{4}^{X2}{{I}^{\prime}}_{2}^{2})$, $\beta =({n}_{2}^{X2}{{I}^{\prime}}_{2}+{n}_{4}^{X2}{{I}^{\prime}}_{2}^{2})+\left|({\Delta}_{1}+{\Delta}_{2})/{\Delta}_{1}\right|[{n}_{2}^{X1}({I}_{1}+{{I}^{\prime}}_{1})+({n}_{2}^{X2}{I}_{2}+{n}_{4}^{X2}{I}_{2}^{2})]$, and ${\gamma}_{m}=m+\mathrm{exp}[-{n}_{2}^{X2}({I}_{2}+{{I}^{\prime}}_{2})-{n}_{4}^{X2}({I}_{2}^{2}+{{I}^{\prime}}_{2}^{2})]$, respectively. It is easy to see that $\alpha \approx 0$ if ${E}_{2}$ is turned off around TPR or far away from TPR ($\Delta {n}_{2}^{X1}$ decreases rather sharply), $\beta \approx 0$ if ${{E}^{\prime}}_{2}$ is blocked around TPR, and

*m*in ${\gamma}_{m}$ is 0 for the dipole mode modulation (Fig. 3(c)) and 1 for modulated vortex mode modulation (Fig. 4). So when ${{E}^{\prime}}_{2}$ is blocked in Fig. 4(a1), fundamental mode soliton is obtained around TPR due to sufficiently strong XPM focusing of ${E}_{2}$. If ${\Delta}_{2}$ is far away from TPR, $\alpha \approx 0$ and $\beta $ should be considered because $\Delta {n}_{2}^{X1}$ does not change, so the non-focusing AMV modes appear as shown in the cases of Figs. 4(a2)-(a5). In other word, ${E}_{F}$ keeps the vortex pattern as compression effect of the XPM nonlinearity from ${{E}^{\prime}}_{2}$ and ${E}_{2}$ is very weak. When ${E}_{2}$ is blocked in Fig. 4(a2), $\alpha \approx 0$ but $\beta \ne 0$ around TPR. Since the second term in ${\gamma}_{m}$ is very small because of the large ${n}_{2}^{X2}{{I}^{\prime}}_{2}+{n}_{4}^{X2}{{I}^{\prime}}_{2}^{2}$, we observe a dipole-like mode around TPR, which is due to the weak focusing effect of ${{E}^{\prime}}_{2}$. When no beam is blocked, as shown in Fig. 4(a3), the total intensity from ${{E}^{\prime}}_{2}$ and ${E}_{2}$ is large enough to balance the competing CQ nonlinearities. In this case, $\alpha <\beta $ because of a higher ${E}_{2}$ field as shown by the middle dot in Fig. 2(b) and ${\gamma}_{m}\approx 2$, so we observe a vortex mode versus different ${\Delta}_{2}$. The essential reason for the transformation among the three types of modes is the competition between third- and fifth-order nonlinearities.

In Figs. 4(a4) and (a5), we increase the power ${{P}^{\prime}}_{2}$ with ${E}_{2}$ fixed, and find that the higher total intensity breaks the balance between the cubic and quintic nonlinearities, and the latter one defocuses the beam. This becomes more obvious around TPR because the magnitude of $\Delta {n}_{4}^{X2}$ increases much more significantly than that of $\Delta {n}_{2}^{X2}$, and therefore the defocusing effect exceeds the focusing effect. The detailed explanations are similar to the ones used for Figs. 3(a1)-(a3) and (c1)-(c3). Figs. 4(b1)-(b3) obtained from Eq. (5), which correspond to Figs. 4(a1)-(a3), visually display that the modulated FWM transform vortex modes into fundamental mode or dipole mode around TPR. In Fig. 4, it is worth noting that the vortex versus ${\Delta}_{2}$ formed here is significantly different from those exhibited in [5,6,7], i.e., the pattern does not rotate compared to the usual vortex solitons. Such stability is due to the counteraction between nonlinear phase shift ${\varphi}_{F}$ and the intrinsic phase from nonzero topology charge and azimuthal modulation in Eq. (5).

## 4. Conclusion

In conclusion, we have experimentally investigated the multi-parameter controllable solitons under competing ${\chi}^{(3)}$ and ${\chi}^{(5)}$ nonlinear susceptibilities enhanced by atomic coherence or EIT in a three-level atomic system. Both in the spatial images and spectral intensity curves of the probe transmission and FWM beams, we have found that the nonlinear propagation processes can be significantly affected by the fifth-order nonlinearity if the pump field intensities are high. In addition, we have demonstrated transformations among fundamental, dipole and AMV solitons by controlling the pump beam geometric configuration and/or competing CQ nonlinearities. The experimental and theoretical results agree with each other very well. Such spatial pattern formation and propagation control can have important applications in all-optical image storage, processing and communication.

## Acknowledgments

This work was supported by the National Basic Research Program of China (2012CB921804) and NSFC (10974151, 61078002, 61078020, 11104214, 61108017), CX12189WL02, and CX12189WL03.

## References and links

**1. **R. W. Boyd, *Nonlinear Optics* (Academic Press, 2008), 3rd ed.

**2. **S. E. Harris, “Electromagnetically induced transparency,” Phys. Today **50**(7), 36–42 (1997). [CrossRef]

**3. **M. Mitchell and M. Segev, “Self-trapping of incoherent white light,” Nature **387**(6636), 858–883 (1997). [CrossRef] [PubMed]

**4. **A. Chong, W. H. Renninger, D. N. Christodoulides, and F. W. Wise, “Airy-Bessel wave packets as versatile linear light bullet,” Nat. Photonics **4**(2), 103–106 (2010). [CrossRef]

**5. **A. S. Desyatnikov and Y. S. Kivshar, “Necklace-ring vector solitons,” Phys. Rev. Lett. **87**(3), 033901 (2001). [CrossRef] [PubMed]

**6. **A. S. Desyatnikov, D. Buccoliero, M. R. Dennis, and Y. S. Kivshar, “Suppression of collapse for spiraling elliptic solitons,” Phys. Rev. Lett. **104**(5), 053902 (2010). [CrossRef] [PubMed]

**7. **S. Lopez-Aguayo, A. S. Desyatnikov, and Y. S. Kivshar, “Azimuthons in nonlocal nonlinear media,” Opt. Express **14**(17), 7903–7908 (2006). [CrossRef] [PubMed]

**8. **Y. Zhang, Z. Wang, Z. Nie, C. Li, H. Chen, K. Lu, and M. Xiao, “Four-wave mixing dipole soliton in laser-induced atomic gratings,” Phys. Rev. Lett. **106**(9), 093904 (2011). [CrossRef] [PubMed]

**9. **Y. S. Kivshar and G. P. Agrawal, *Optical Solitons: From Fibers to Photonic Crystals* (Academic Press, 2003).

**10. **C. Rotschild, O. Cohen, O. Manela, M. Segev, and T. Carmon, “Solitons in nonlinear media with an infinite range of nonlocality: first observation of coherent elliptic solitons and of vortex-ring solitons,” Phys. Rev. Lett. **95**(21), 213904 (2005). [CrossRef] [PubMed]

**11. **A. S. Desyatnikov, D. Buccoliero, M. R. Dennis, and Y. S. Kivshar, “Necklace-ring vector solitons,” Phys. Rev. Lett. **104**, 053902 (2010). [CrossRef] [PubMed]

**12. **Y. V. Kartashov, L. C. Crasovan, D. Mihalache, and L. Torner, “Robust propagation of two-color soliton clusters supported by competing nonlinearities,” Phys. Rev. Lett. **89**(27), 273902 (2002). [CrossRef] [PubMed]

**13. **J.-S. Lee, M. V. Kovalenko, J. Huang, D. S. Chung, and D. V. Talapin, “Band-like transport, high electron mobility and high photoconductivity in all-inorganic nanocrystal arrays,” Nat. Nanotechnol. **6**(6), 348–352 (2011). [CrossRef] [PubMed]

**14. **S. Palomba, S. Zhang, Y. Park, G. Bartal, X. Yin, and X. Zhang, “Optical negative refraction by four-wave mixing in thin metallic nanostructures,” Nat. Mater. **11**(1), 34–38 (2011). [CrossRef] [PubMed]

**15. **J. M. Nunzi and F. Charra, “Picosecond two-photon absorption effects on index variation gratings in polydiacetylenes,” Nonlinear Opt. **2**, 131 (1992).

**16. **B. Lawrence, W. E. Torruellas, M. Cha, M. L. Sundheimer, G. I. Stegeman, J. Meth, S. Etemad, and G. Baker, “Identification and role of two-photon excited states in a pi -conjugated polymer,” Phys. Rev. Lett. **73**(4), 597–600 (1994). [CrossRef] [PubMed]

**17. **G. Boudebs, S. Cherukulappurath, H. Leblond, J. Troles, F. Smektala, and F. Sanchez, “Experimental and theoretical study of higher-order nonlinearities in chalcogenide glasses,” Opt. Commun. **219**(1–6), 427–433 (2003). [CrossRef]

**18. **S. Reineke, F. Lindner, G. Schwartz, N. Seidler, K. Walzer, B. Lüssem, and K. Leo, “White organic light-emitting diodes with fluorescent tube efficiency,” Nature **459**(7244), 234–238 (2009). [CrossRef] [PubMed]

**19. **H. Michinel, M. J. Paz-Alonso, and V. M. Pérez-García, “Turning light into a liquid via atomic coherence,” Phys. Rev. Lett. **96**(2), 023903 (2006). [CrossRef] [PubMed]

**20. **Á. Paredes, D. Feijoo, and H. Michinel, “Coherent cavitation in the liquid of light,” Phys. Rev. Lett. **112**(17), 173901 (2014). [CrossRef] [PubMed]

**21. **M. Artoni and G. C. La Rocca, “Optically tunable photonic stop bands in homogeneous absorbing media,” Phys. Rev. Lett. **96**(7), 073905 (2006). [CrossRef] [PubMed]

**22. **Y. Zhang and M. Xiao, *Multi-Wave Mixing Processes: From Ultrafast Polarization Beats To Electromagnetically Induced Transparency *(H & Springer, 2009).

**23. **Y. Zhang, Z. Nie, H. Zheng, C. Li, J. Song, and M. Xiao, “Electromagnetically induced spatial nonlinear dispersion of four-wave mixing,” Phys. Rev. A **80**(1), 013835 (2009). [CrossRef]

**24. **Y. Zhang, Z. Nie, Y. Zhao, C. Li, R. Wang, J. Si, and M. Xiao, “Modulated vortex solitons of four-wave mixing,” Opt. Express **18**(11), 10963–10972 (2010). [CrossRef] [PubMed]

**25. **J. Leach, M. R. Dennis, J. Courtial, and M. J. Padgett, “Laser beams: knotted threads of darkness,” Nature **432**(7014), 165 (2004). [CrossRef] [PubMed]