现在写这类题材的应该不多了，毕竟非为言志，格调不高，难登大雅。且集中所用词句，多为陈词滥调，并无太多新意，唯独韵是新的。所描所载，皆为虚假，如镜花水月、梦幻泡影，概古人情事之再现也。

《琵琶》

银屏琵琶冷如霜，一片眸深半月光。

雁去痴情红豆撒，相逢黄叶入秋窗。

《琴丝》

游园一曲落深红，指上银钩柱上浓。

莫道琴丝伤寸许，春宵宴罢几人重?

《枯柳》

梧桐井底柳千条，莫忆双成待玉霄。

但乞枯枝零落远，江流一叶伴君艄。

《蝴蝶》

明暗蝴蝶度绿丝，远山眉黛小桃枝。

行囊绵里针头密，绮珞沉箱客不知。

《画屏》

寂静空楼月上迟，去年今夜画屏时。

同生莲子心相印，根茎荷花两不知。

《春衫》

瓦上秋霜月上弦，佳期过梦似流年。

春衫不改同心扣，故作初时共枕眠。

《椒房》

一段白绫万念休，当年鬓鬒绕垂旒。

椒房不似金屋冷，碧玉清辉落上头。

《花窗二首、其一》

小筑花窗夜未休，明灯一盏缀金裘。

懒起清风撩鬓角，话别枝鹊吊眉头。

《花窗二首、其二》

莫道君心夜夜心，花笺旧句未传情。

西门向使城头过，遥见窗前一盏明。

《晓山》

故事纷杂梦已轻，鸡声残月晓山行。

霜花不必春花赏，败叶多于绿叶情。

《春庭草》

春庭草，秋宫叶。似欢声，如笑靥。

声去铃铃金络脑，眼前忽见乌丝屧。

春池藻，秋雨潦，飘蓬一缕孤魂老。

云边征雁向南飞，迢迢无处衣鲁缟。

《燕雀》

燕雀相期入画梁，金屋无赖怨辽阳。

霜梨红枣牵丝短，翠玉珍珠寡味长。

《无题》

迢递星辰暗玉台，桂堂心字半成灰。

可怜香断根犹在，此夜风窗落絮堆。

《所历》

渡外江天远，寒山客路长。

疾风归越鸟，孤棹入斜阳。

月起昏鸦晚，帆升暮翳翔。

船夫询就所，落魄在他乡。

《雁门关》

风劲鬼哭月如斗，黑烟扑面成苍狗。

两峰徘徊孤雁出，相随一骑滚石走。

雁过山河表里宁，尸横塞上膏野韭。

天京城下洞门开，此地持节凭太守。

辗转经年岁，残梦五更歇。

皓月旧相识，幽窗半掩斜。

Afrer installation, how to select the mirrors by country and update your system?

sudo pacman-mirrors --country Singapore
sudo pacman -Syyu
sudo pacman -S base-devel # Install basic developing tools

How to config for Chinese input?

sudo pacman -S fcitx5-chinese-addons fcitx5-gtk fcitx5-qt
sudo pacman -S fcitx5-configtool
touch ~/.pam_environment
echo 'GTK_IM_MODULE=fcitx5
QT_IM_MODULE=fcitx5
XMODIFIERS=@im=fcitx5' >> ~/.pam_environment
# echo fcitx5 configuration Finished

Is there any pacman tricks?

sudo pacman -S yay  ## Use AUR repository
sudo pacman -Rns $(pacman -Qtdq)  ## Pacman Clean
sudo pacman -Scc

How to let software run as backstage process? Signal as an example

# sudo pacman -S signal-desktop
if [ -f "~/.signal_history" ]; then
     rm ~/.signal_history
fi
nohup signal-desktop --start-in-tray --use-tray-icon > ~/.signal_history 2>&1 &

How to add environment path? Texlive as an example

## add to path in /etc/profile
export PATH=$PATH:/usr/local/texlive/20xx/
export PATH=$PATH:/usr/local/texlive/20xx/bin/xxxx
export PATH=$PATH:/usr/local/texlive/20xx/texmf-dist/doc/man
export PATH=$PATH:/usr/local/texlive/20xx/texmf-dist/doc/info

How to connect host with SSH? Github as an example

# (1) Check current ssh-key
ls -al ~/.ssh
# Now we have private key "id_rsa" and public key "id_rsa.pub"
# (2) Create a key
ssh-keygen -C "zhuanglinsheng@outlook.com"
# (3) Add ssh-key to ssh-agent
## First, start ssh-agent
eval "$(ssh-agent -s)"
# Then, add private key to agent
ssh-add ~/.ssh/id_rsa
# (4) Copy public key to github: Settings > SSH and GPG Keys > SSH Keys
cat ~/.ssh/id_rsa.pub
# (5) Test ssh connection
ssh -T git@github.com
# (6) Sync repository: using git@github.com:user-name/repository-name.git
#     instead of https://github.com/user-name/repository-name.git
# (7) How about SSHD?
sudo systemctl enable sshd.service
sudo systemctl start sshd.service
sudo systemctl status sshd.service
sudo systemctl stop sshd.service

How to make your computer a server? Using ngrok

## Install Ngrok from AUR:
yay install ngrok
## Verification (login https://ngrok.com/ and find token):
ngrok authtoken TOKEN
## Ngrok proxy (Usages)
if [ -f "~/.nohup_history" ]; then
    rm ~/.nohup_history
fi
nohup ngrok tcp 22 --region ap -log=stdout > ~/.nohup_history 2>&1 &
ps -aux | grep "ngrok"
## Find the ngrok port in '.nohup_history' file (and it shows to be xxxxx = a number)
cat ~/.nohup_history | grep "url"
## General ssh Connection
ssh USER@0.tcp.ap.ngrok.io -p xxxxx

## App: Jupyter
ssh -N -L localhost:8080:localhost:8888 USER@0.tcp.ap.ngrok.io -p xxxxx
## In browser: localhost: 8888. Type in the default jupyter password.
## App: Rstudio-server
ssh -N -L localhost:8181:localhost:8787 USER@0.tcp.ap.ngrok.io -p xxxxx
## In browser: localhost: 8181. Login as USER.

Is there any pip tricks?

# (1) Install packages
sudo pacman -S python-pip
pip3 install numpy scipy pandas matplotlib
# (2) pip installed packages' location:
# ~/.local/lib/python3.8/site-packages/
# (3) pip operations: list outdated
pip list -o
# (4) pip operations: updated outdated
pip list -o -f=freeze | grep -v '^\-e' | cut -d = -f 1 | xargs -n1 pip install -U

How can I use jupyter remotely?

# Install jupyter notebook extensions
pip install jupyter_contrib_nbextensions
jupyter contrib nbextension install --user --skip-running-check
# Basic jupyter environment settings
## Set password, the same as user password:
jupyter notebook password
# Start Jupyter Service
if [ -f "~/.jupyter_history" ]; then
    rm ~/.jupyter_history
fi
nohup jupyter notebook --no-browser --port=8888 > ~/.jupyter_history 2>&1 &
ps -aux | grep "jupyter"

晨雨稀疏碎叶轻，混入芳菲散若云。

红楼今时如往日，满纸冰心作妄心。

——黛玉

无边秋雨肃秋尘，一地秋花散妍痕。

随风从此飘零去，除却曾经几缕魂。

Reference: NUS MA6253 Conic Programming Lecture Notes of Sun Defeng.

1. Metric Projection

Def. (Metric Projection) For any set $K$ that is closed and convex in $H$ and for any $z\in H$, let

This optimization problem has a unique solution.

Note. For all $d\in K$ we have $\langle z - \Pi_K(z),d-\Pi_K(z)\rangle \le 0$.

Proof.

For any $d\in K$, letting $y_t=(1-t)\Pi_K(z)+td$ for any $t\in[0,1]$, then

since $y_t\in K$ by the convexity of $K$. Thus,

Letting $t\to0$ and we have

Note. If $K$ is a cone, then for any $d\in K$ we have

The first equality is by $0\in K$ and $2\Pi_K(z)\in K$. The second inequality is by $d+\Pi_K(z)\in K$.

2. Relative Interior

Def. (Relative Interior) For any convex set $\mathcal{C}\subset R^n$, the relative interior of $\mathcal{C}$ denoted by $\text{ri}(\mathcal{C})$ is defined by

where $\text{aff}(\mathcal{C})$ is the affine hull of $\mathcal{C}$.

Lemma 1. Let $\mathcal{C}$ be a convex set in $R^n$, let $x\in\text{ri}(\mathcal{C})$ and $y\in cl(\mathcal{C})$. Then $(1-\alpha)x + \alpha y \in \text{ri}(\mathcal{C})$ for all $\alpha\in[0,1)$.

Proof.

For any given $\alpha \in [0,1)$, denote by $m \equiv (1-\alpha)x + \alpha y$. We want to show that $B_\epsilon(m)\cap \text{aff}(\mathcal{C})\subset \mathcal{C}$ for some $\epsilon>0$. Since $x\in \text{ri}(\mathcal{C})$ we know there exists $\epsilon_x > 0$ such that $B_{\epsilon_x}(x)\cap\text{aff}(\mathcal{C}) \subset \mathcal{C}$. For all $x’ \in B_{\epsilon_x}(x)$ we have $(1-\alpha)x’ + \alpha y \in cl(\mathcal{C})$ by the convexity of $\mathcal{C}$, which implies $B_{(1-\alpha)\epsilon_x}(m) \subset cl(\mathcal{C})$. Since $\alpha < 1$ we have $(1-\alpha) \epsilon_x > 0$, and we simply pick $\epsilon = (1-\alpha) \epsilon_x / 2 > 0$ and $B_{\epsilon}(m) \subset \mathcal{C}$. Finally, $B_{\epsilon}(m) \cap \text{aff}(\mathcal{C}) \subset \mathcal{C}$. END.

Lemma 2. For any convex set $\mathcal{C}$ in $R^n$, we have $cl(\text{ri}(\mathcal{C})) = cl(\mathcal{C})$ and $\text{ri}(cl(\mathcal{C})) = \text{ri}(\mathcal{C})$.

Proof.

(1) All we need to show is that $cl(\mathcal{C}) \subset cl(\text{ri}(\mathcal{C}))$. For any $y\in cl(\mathcal{C})$, pick $x\in ri(\mathcal{C})$. By Lemma 1 we have

where $a_n \in (0,1]$ and $a_n \to 1$. It is obvious that $y_n \to y$. Thus, $y \in cl(\text{ri}(\mathcal{C}))$.

(2) All we need to show is that $\text{ri}(cl(\mathcal{C})) \subset \text{ri}(\mathcal{C})$. For any $y\in \text{ri}(cl(\mathcal{C}))$, we have $\epsilon_y > 0$ such that $B_{\epsilon_y}(y) \cap \text{aff}(cl(\mathcal{C})) \subset cl(\mathcal{C})$. Firstly, we want to show that $\text{aff}(cl(\mathcal{C})) = \text{aff}(\mathcal{C})$ by showing that $\text{aff}(cl(\mathcal{C})) \subset \text{aff}(\mathcal{C})$. for any $y \in \text{aff}(cl(\mathcal{C}))$ we can pick two points $x, z \in cl(\mathcal{C})$ such that $y = \alpha x + (1-\alpha)z$. Then we pick two sequences $(x_n)\in \mathcal{C}$ and $(z_n) \subset \mathcal{C}$ such that $x_n\to x$ and $z_n\to z$, and define $y_n\equiv \alpha x_n + (1-\alpha) z_n \in \text{aff}(\mathcal{C})$. Then $y_n \to y$. By the completeness of $\text{aff}(\mathcal{C})$ we have $y\in \text{aff}(\mathcal{C})$. Secondly, by selecting $\epsilon= \epsilon_y / 2$ and we have $B_\epsilon(y) \subset \mathcal{C}$. Finally, we conclude that $B_\epsilon(y) \cap \text{aff}(\mathcal{C}) \subset \mathcal{C}$, and then $y\in \text{ri}(\mathcal{C})$.

3. Fréchet Differentiation

Def. (Fréchet Differentiation) A function $f:\mathcal{X}\to\mathcal{Y}$ is (Fréchet) differentiable at $x\in\mathcal{X}$ if there exists a linear operator $f’:\mathcal{X}\to\mathcal{Y}$ such that for any $h\to 0$ we have

Let $\nabla f(x)$ be the adjoint of $f’(x)$.

Lemma 3. (Regularity of Fréchet differential) If the Fréchet derivative of $h:\mathcal{X}\to\mathcal{Y}$ at $\bar{x}$ is onto, then there exists a $M>0$ such that for all $y\in\mathcal{Y}$ there exists a $d\in\mathcal{X}$ with $y = h’(\bar{x})d$ and $\Vert d \Vert \le M\Vert y \Vert$.

Proof.

The condition that $h$ is onto is equivalent to that $0 \in \text{int}(h’(\bar{x})\mathcal{X}) = \text{int}(\text{range}(h’(\bar{x})))$. By Proposition 2 of Perturbation of Correspondence we have $x\in \text{null}(h’(\bar{x}))$,

Equivalently, there exists $\eta>0$ such that

Thus, for all $y\in\mathcal{Y}$ there exists $v$ with $|v|\le 1$ and

Then it is easy to verify that there exists $M>0$ such that $\Vert d \Vert \le M\Vert y \Vert$.

4. liminf and limsup of Correspondence

Def. (liminf and limsup) The upper and lower limits of a set map (parameterized family) $A_t\subset\mathcal{Y}$ are

Lemma 4. (Closeness of liminf and limsup) $\limsup_{t\to t_0} A_t$ and $\liminf_{t\to t_0} A_t$ are closed.

Proof.

Denote by $\bar{A_t}(t_0)\equiv\limsup_{t\to t_0} A_t$ and $\underline{A_{t}}(t_0)\equiv\liminf_{t\to t_0} A_t$.

(1) For any sequence $(a_n)\subset\bar{A_t}(t_0)$ with $a_n\to a$, since $a_n\in \bar{A_t}(t_0)$ we know $\exists (t^n_k)$ with $t^n_k\to t_0$ s.t. $\exists y^n_k\in A_{t^n_k}$ with $y^n_k\to a_n$. For ant fixed $n$, define

Let $t_n = t^n_{k_n}$ and $y_n = y^n_{k_n} \in A_{t_n}$. We have $t_n\to t_0$ and

Thus, $a\in\bar{A_t}(t_0)$ and $\bar{A_t}(t_0)$ is closed.

(2) For any sequence $(a_n)\subset \underline{A_{t}}(t_0)$ with $a_n\to a$ and for any sequence $t_k\to t_0$, since $a_n\in\underline{A_{t}}(t_0)$ we know $\exists y^n_k\in A_{t_k}$ with $y^n_k\to a_n$. Let $k_1 = 1$. For $n\ge2$, define a sequence $(k_n)$ by

It’s obviously that $(k_n)$ is increasing. Define $(y_k)$ in the following recursive way:

Step 1. Let $n=1$.

Step 2. For all $k_n \le k < k_{n+1}$, let $y_k = y^n_k$. Otherwise, go to step 3.

Step 3. let $n$ be $n+1$. Go to Step 2.

It is obviously that $y_k \in A_{t_k}$ for all $k$. Now we show that $y_k\to a$. For any $k_{n} \le k < k_{n+1}$ we have

Thus, we conclude that $a\in\underline{A_{t}}(t_0)$ and $\underline{A_{t}}(t_0)$ is closed.

5. Tangent Cones

Def. (Tangent Cones) For any closed set $D\subset \mathcal{Y}$ and a point $y\in D$ we define

(1) Radial cone

(2) Inner tangent cone

(3) Contingent (Bouligand) cone

(4) Clarke tangent cone

Lemma 5. (Sequential Representations of Tangent Cones) For inner tangent cone and Bouligand cone, we have the following sequential representations

Proof.

We only show Bouligand tangent cone, the proof of the other one is similar.

”$\subset$”. By definition of “limsup” we know for all $d\in T_D(y)$, $\exists$ $(t_k)$ with $t_k\downarrow 0$ and $\exists d_k\in \frac{D-y}{t_k}$ such that $d_k\to d$, meaning that $y+t_kd_k\in D$ with $d_k\to d$. Thus, we have

Thus, we conclude that $D_\mathcal{Y}(y+t_kd, D) = o(t_k)$.

”$\supset$”. For all $d\in\mathcal{Y}$ such that $\exists (t_k)$ with $t_k\downarrow 0$ and $D_\mathcal{Y}(y+t_kd,D)=o(t_k)$, meaning that $\exists (x_k)\subset D$ with

Let $d_k = \frac{x_k-y}{t_k}\in\frac{D-y}{t_k}$ and we have

which implies $d_k\to d$. Thus, $d\in T_D(y)$.

Lemma 6. (Relations among Tangent Cones 1) From the definitions above we directly have

(1) $T^i_D(y)$, $T_D(y)$ and $T^c_D(y)$ are all closed, and

(2) $R_D(y)\subset T^i_D(y)\subset T_D(y)$.

Proof.

(1) By the property of “limsup” and “liminf” (see Lemma 4).

(2) It is obviously that $T^i_D(y)\subset T_D(y)$. Now we want to show that $R_D(y)\subset T^i_D(y)$. For any $d\in R_D(y)$, by definition we know there exists a $t^\star>0$ such that $y+td\in D$ for all $t\in [0,t^\star]$. For any $(t_k)$ with $t_k\downarrow 0$, there exists $K>0$ such that $t_k < t^\star$ when $k>K$. Then we have $y+t_kd\in D$, and $D_\mathcal{Y}(y+t_kd,D) = 0 = o(t_k)$. Thus, we conclude that $d\in T^i_D(y)$.

Lemma 7. If $y\not\in D$ then $ R_D(y) = \emptyset$. Same for $T^i_D(y)$ and $T_D(y)$ (directly from the sequential representation).

Lemma 8. (Relations among Tangent Cones 2) If $D\subset\mathcal{Y}$ is closed and convex and $y\in D$, then we have

(1) $R_D(y) = \cup_{t>0}(D-y)/t$.

(2) $T_D(y) = T^i_D(y) = cl(R_D(y))$.

Proof.

(1) The “$\subset$” direction is straightforward. Now we want to show the “$\supset$” direction. For any $d\in \cup_{t>0}(D-y)/t$ there exists $t^\star>0$ and $s\in D$ such that $d = (s-y)/t^\star$. Then for all $t\in[0,t^\star]$ we have

which implies that $d\in R_D(y)$.

(2) First we have $R_D(y)\subset T^i_D(y)\subset T_D(y)$ by Lemma 6 and $T^i_D(y), T_D(y)$ are closed by Lemma 6. Thus we have

Conversely, for any $d\in T_D(y)$, by definition there exists $(t_k)\downarrow 0$ and $(y_k)\subset D$ such that $(y_k-y)/t_k\to d$. Since $(y_k-y)/t_k\in R_D(y)$ by the conclusion of Part (1) we have $d\in cl(R_D(y))$.

Def. (Normal Cone) For any closed $D\subset\mathcal{Y}$, the normal cone to $D$ at $y\in\mathcal{Y}$ is defined by

i.e., the polar cone of Bouligand cone of $D$ at $y$.

Lemma 9. $N_D(y)$ is closed since polar cone is always closed.

Lemma 10. $N_D(y) = \emptyset$ if $y\not\in D$ since polar cone of empty set is empty set. (There are different definitions of polar of empty set and here we follow Sun’s notes.)

Lemma 11. If $D$ is closed and convex and $y\in D$, remember we have $R_D(y) = \cup_{t>0}(D-y)/t$ and then

6. The Optimization Problem

Def. (Optimization Problem, OP) Suppose $f, g, h$ are $C^1$ (in Fréchet differentiable sense) and $\mathcal{C}$ is a closed and convex set. Then the optimization problem (OP) is defined by

Def. (Compact Optimization Problem, COP) As a generalization [by letting $G = (h,g)$ and $\mathcal{K}={0}^m\times\mathcal{C}$ ], the compact form of OP (COP) is defined as

where $f:\mathcal{X}\to R$ and $G:\mathcal{X}\to\mathcal{Y}$ are $C^1$ (in Fréchet differentiable sense) and $\mathcal{K}\subset\mathcal{Y}$ is a closed and convex set. Let

be the feasible set and

be the Lagrangian, where $\mu\in\mathcal{Y}^*$.

Def. (Lagrange Multiplier and KKT Conditions) We say $\bar{\mu}$ is a Lagrange multiplier of COP at feasible point $\bar{x}$ if it satisfies the KKT conditions

Lemma 12. (KKT Conditions when $\mathcal{K}$ is a Cone) If $\mathcal{K}$ is a closed and convex cone, then the KKT condition above is equivalent to

Proof.

Since $\bar{x}$ is feasible we have $G(\bar{x})\in\mathcal{K}$. Thus, by Lemma 11 we have

Thus, $-\bar{\mu}\in N_\mathcal{K}(G(\bar{x}))$ implies $\langle \bar{\mu},G(\bar{x}) \rangle \le \langle \bar{\mu},u \rangle$, $\forall u\in\mathcal{K}$. Since $\mathcal{K}$ is a close convex cone, we have $0\in\mathcal{K}$ and $2G(\bar{x})\in\mathcal{K}$, thus

Finally we conclude that $\langle \bar{\mu},G(\bar{x}) \rangle = 0$. Further more,

implies that $\bar{\mu}\in\mathcal{K}^\star$.

Example 1. (Bouligand cone of OP) Let $\mathcal{K} = {0}^m\times R^q_+$ and

the Bouligand cone $T_\mathcal{K}(G(\bar{x}))$ is

where $d^1\in R^m,d^2\in R^q$. Thus, for any $(d_1^T,d_2^T)^T\in T_\mathcal{K}(G(\bar{x}))$ there exists some $t^k\downarrow0$ such that

and

Define $I(\bar{x})\equiv{i:g_i(\bar{x})=0}$ and we know that for all $i\in I(\bar{x})$ we have

and for all $i\not\in I(\bar{x})$ we have

which holds for all $d^2_i$, $i\not\in I(\bar{x})$. Finally,

7. Robinson’s CQ: Basic Concepts

Def. (Robinson’s CQ) Robinson’s CQ (RCQ) holds at $\bar{x}$ if

Lemma 13. Here $G’(x)$ is the linear operator from $\mathcal{X}$ to $\mathcal{Y}$ representing the Fréchet derivative of $G(x)$.

Lemma 14. Since $G’(x)$ is a linear operator, we have $G’(x)\mathcal{X}$ is a linear subspace of $\mathcal{Y}$.

Lemma 15. (Polar Cone Algebra) For any cones $A$ and $B$ we have

Proof.

(1) “$\subset$”. For any $d\in [A+B]^\circ$ we must have $d\in A^\circ$ since $0\in B$ and $d\in B^\circ$ since $0\in A$.

(2) “$\supset$”. For any $d\in A^\circ \cap B^\circ$ we must have $\langle d,a+b\rangle = \langle d,a\rangle + \langle d,b\rangle \le 0$ for all $a\in A$ and $b\in B$.

Proposition 1. (RCQ at feasible $\bar{x}$) Assume $G(\bar{x})\in\mathcal{K}$, then the following expressions (a)-(e) are equivalent, and are equivalent to RCQ,

Proof.

$(a)\Rightarrow(b)$. By Lemma 6, it’s obvious.

$(b)\Leftrightarrow(c)$. Compute the polar cone of (b) on both sides and we have

The third equality is by the Lemma of Polar Cone Algebra. The last equality holds because $G’(x)\mathcal{X}$ is a linear subspace. is feasible.

$(c)\Rightarrow(d)$. Using (b) to derive (d) since (b) is equivalent to (c).

$(d)\Rightarrow(e)$. Given that $G(x)\in\mathcal{K}$ and $K$ is convex and closed, we calculate the polar cone of (d)​ on both sides and get

which implies (e). The forth equality is by Lemma 8.

$(e)\Rightarrow(a)$. If (a) fails to hold then $\text{ri}[G’(\bar{x})\mathcal{X} + R_\mathcal{K}(G(\bar{x}))] \not= \mathcal{Y}$. Then by (e) we have a contradiction,

$\text{RCQ}\Rightarrow(a)$. Given that $G(x)\in\mathcal{K}$ and $\mathcal{K}$ is convex and closed, for any $y\in\mathcal{Y}$, by RCQ we know there exists some $t>0$ such that

which implies that $\mathcal{Y} \subset G’(\bar{x})\mathcal{X}+\mathcal{R}_\mathcal{K}(G(\bar{x}))$.

$(a)\Rightarrow\text{RCQ}$. Define a correspondence $\mathcal{M}:\mathcal{X}\times[0,\infty)\rightrightarrows\mathcal{Y}$ by

Since $\mathcal{K}$ is convex and closed, we know that $\mathcal{M}$ is convex and closed. By (a) we have

which implies

By Proposition 2 of Perturbation of Correspondence, for any $(x,t)\in\mathcal{M}^{-1}(0)$ we have

Since $(0,0)\in\mathcal{M}^{-1}(0)$, the above condition is transformed into

The last equality is because $\mathcal{K}-G(\bar{x})$ is convex and closed and contains the origin, thus $t(\mathcal{K}-G(\bar{x}))$ is the shrink of $\mathcal{K}-G(\bar{x})$ when $t\in[0,1]$, and thus contained by $\mathcal{K}-G(\bar{x})$.

Proposition 2. (RCQ at feasible $\bar{x}$ for solid $\mathcal{K}$) Assume $G(\bar{x})\in\mathcal{K}$ and $\mathcal{K}$ has a nonempty interior, RCQ is equivalent to $G(\bar{x})+G’(\bar{x})d\in \text{int}(\mathcal{K})$ for some $d\in\mathcal{X}$.

Proof.

$\Leftarrow$. If such a $d\in\mathcal{X}$ exists then there exists $\epsilon > 0$ such that

Thus the RCQ holds.

$\Rightarrow$. Suppose that for all $d\in\mathcal{X}$ we have $G(\bar{x})+G’(\bar{x})d\not\in \text{int}(\mathcal{K})$. Then we know the sets $G(\bar{x})+G’(\bar{x})\mathcal{X}$ and $\text{int}(\mathcal{K})$ are isolated. Then by separation theorem there exists $\lambda\in\mathcal{Y}^*$ such that

for all $d\in\mathcal{X}$ and $k\in\mathcal{K}$. Thus, for any $y\in\mathcal{Y}$ such that $\langle\lambda,y\rangle<0$ we know $ty\not\in G(\bar{x})-G’(\bar{x})\mathcal{X}-\mathcal{K}$ for all $t>0$. Thus $B(0,\epsilon)\not\subset G(\bar{x})-G’(\bar{x})\mathcal{X}-\mathcal{K}$ for all $\epsilon>0$.

8. Robinson’s CQ and Other CQs

Proposition 3. (RCQ for OP) Suppose $G(x)=(h(x),g(x))$, $\mathcal{K}={0}^m\times\mathcal{C}$ and $G(\bar{x})\in\mathcal{K}$. COP becomes

where $f,h,g$ are Fréchet differentiable. If $\mathcal{C}$ is closed, convex and has a nonempty interior, then RCQ at the feasible point $\bar{x}$ is equivalent to

(A) $h’(\bar{x})$ is onto;

(B) $\exists d\in\mathcal{X}$ such that $h’(\bar{x})d=0$ and $g(\bar{x})+g’(\bar{x})d\in \text{int}(\mathcal{C})$.

Proof.

”$\Rightarrow$”. Since RCQ holds, we know there exists $\epsilon_1>0$ and $\epsilon_2>0$ such that for all $y \in B_\mathcal{Y}(\epsilon_1)$ and $z \in B_\mathcal{Y}(\epsilon_2)$, there exists $d\in\mathcal{X}$ with

(A) $y=h(\bar{x})+h’(\bar{x})d-{0}^m = h’(\bar{x})d$

(B) $z\in g(\bar{x})+g’(\bar{x})d-\mathcal{C}$

Condition (A) implies that $h’(\bar{x})$ is onto. Letting $y=0$ we have $d\in \text{null}(h’(\bar{x}))$, implying that

Thus, there exists $d\in\mathcal{X}$ such that $0=h’(\bar{x})d$ and $g(\bar{x})+g’(\bar{x})d\in \text{int}(\mathcal{C})$.

”$\Leftarrow$”. Since $\text{int}(\mathcal{C})\not=\emptyset$, the conditions above is equivalent to

(A) $h’(x)$ is onto,

(B) $0\in \text{int}(g(\bar{x})+g’(\bar{x})(\text{null}(h’(\bar{x}))-\mathcal{C})$.

By Lemma 3, condition (A) is equivalent to that there exists a constant $M>0$ such that for all $y\in\mathcal{Y}$ there exists a $d_1\in\mathcal{X}$ with $y = h’(\bar{x})d_1$ and $|d_1| \le M|y|$.

Condition (B) is equivalent to $B(0,\epsilon)\subset g(\bar{x})+g’(\bar{x})(\text{null}(h’(\bar{x}))-\mathcal{C}$ for some $\epsilon>0$.

RCQ holds if there exists $\epsilon_1>0$ and $\epsilon_2>0$ such that for all $y \in B_\mathcal{Y}(\epsilon_1)$ and $z \in B_\mathcal{Y}(\epsilon_2)$, there exists $d\in\mathcal{X}$ with

Let $d_2 = d-d_1$, the above conditions are transformed into finding such a $d_2$ that

By letting $\epsilon_1$ and $\epsilon_2$ sufficiently small we have $z-g’(\bar{x})d_1\in B(0,\epsilon)$. Thus, there exists $d_2$ satisfying the above two conditions.

Proposition 4. (RCQ for NLP: MFCQ) For the NLP below,

where $f,g,h$ are all smooth. RCQ is equivalent to MFCQ:

(A) $(\nabla h_i(\bar{x}))^m_{i=1}$ are linearly independent,

(B) $\exists \bar{d}\in\mathcal{X}$ such that $\langle \nabla h_i(\bar{x}),\bar{d}\rangle=0$, $i=1,…,m$ and $\langle\nabla g_j(\bar{x}),\bar{d}\rangle>0$, $j\in \mathcal{I}(\bar{x})$.

where $\mathcal{I}(\bar{x})$ is the index set of $g_i(.)$ that is active at $\bar{x}$.

Proof.

Let

By Proposition 1(b) we have RCQ is equivalent to

where $T_\mathcal{K}(G(\bar{x}))$ is given in Example 1. Hence, RCQ is transformed into

where

“RCQ\(\Rightarrow\)MFCQ”. By RCQ, $h’(\bar{x})\mathcal{X}=R^m$ implies that $h’(\bar{x})$ is linearly independent. For the point $(0, …, 0, -1, …, -1)\in R^{m+q}$ where the first $m$ scalers are zero and remaining are -1, by RCQ there must be a $d\in\mathcal{X}$ such that $h’(\bar{x})d=0$ and $v_j\ge0$, $\nabla g_j(\bar{x})d+v_j=-1$ where $j\in I(\bar{x})$. Thus, $\nabla g_j(\bar{x})d<0$ for $j\in I(\bar{x})$. Then $\bar{d}=-d$ is what we need.

“MFCQ\(\Rightarrow\)RCQ”. Condition (A) of MFCQ implies that $h’(\bar{x})\mathcal{X}=R^m$. Suppose there exists a $\bar{d}\in\mathcal{X}$ such that condition (B) of MFCQ holds, then for any $\begin{bmatrix}y_1 \ y_2\end{bmatrix}\in R^{m+q}$ where $y_1\in R^m$ and $y_2\in R^q$, we have

(A) there exists a $d\in\mathcal{X}$ such that $h’(\bar{x})d = y_1$.

(B) there exists some $k<0$ and $v\in{v\in R^q:v_j\ge0\text{ if }j\in I(\bar{x})}$ such that $y_2 - g’(\bar{x})d = \langle g’(\bar{x}),k\bar{d}\rangle+ v$.

Thus,

which implies that RCQ holds.

9. Approximation of $\Psi_G$ at $(\bar{x}, 0)$

Suppose that $\bar{x}$ is a feasible point where the Robinson’s CQ holds. Denote by $\Psi_H(x) \equiv H(x) - \mathcal{K}$ and we have

Note that $\mathcal{K}$ is closed and convex and so is $\Psi_H$. By the Proposition 3 of Perturbation of Correspondence we have $\Psi_H$ is open at $(\bar{x},0)$ at the rate of $\gamma$ for some $\gamma>0$. Then by Proposition 4 of Perturbation of Correspondence we know $\Psi_H$ is metric regular at $(\bar{x},0)$ at the rate $1/\gamma$.

Define $H(x) \equiv G(\bar{x})+G’(\bar{x})(x-\bar{x})$ the first order Taylor approximation of $G$ around $\bar{x}$. Since $G$ is (Fréchet) continuously differentiable, we know

is continuous. It is straightforward that there $A$ is locally Lipschitz, and there exists a neighborhood of $\bar{x}$ with the Lipschitz modulus $\kappa < \gamma$.

Finally, by the Proposition 5 of Perturbation of Correspondence we know $\Psi_G(x) \equiv G(x) - \mathcal{K}$ is metric regular at the point $(\bar{x},0)$ at the rate $1/(\gamma-\kappa)$.

Proposition 5. For COP, if Robinson’s CQ holds at $\bar{x}$ then there exists a neighborhood $\mathcal{N}$ of $\bar{x}$ such that for all $x\in\mathcal{N}$ we have

Proof.

Since Robinson’s CQ holds at $\bar{x}$ we know $\Psi_G$ is metric regular at $(\bar{x},0)$. Then there exists a neighborhood $\mathcal{N}$ of $(\bar{x},0)$ and a $c>0$ such that

for all $(x, y)\in \mathcal{N}$. Note that

We have

By letting $y = 0$ we have the result.

10. Linearization of COP

Def. (Linearization of COP) The linearization of COP at $\bar{x}\in\mathcal{X}$ is given by

Proposition 6. (Optimal Solution of LCOP) Let $\bar{x}$ be a locally optimal solution of COP. Then $d=0$ is an optimal solution of LCOP.

Proof.

For any $d\in T_\mathcal{F}(\bar{x})$. By the definition of Bouligand tangent cone, there exists a sequence $t^k\downarrow 0$ and $(x^k)\subset\mathcal{F}$ such that $x^k = \bar{x}+t^kd+o(t^k)$. Since $\bar{x}$ is locally optimal solution of COP we have

Also, $0\in T_\mathcal{F}(\bar{x})$, meaning that $d=0$ is feasible.

Proposition 7. (RCQ for LCOP) Suppose $\bar{x}$ is a locally optimal solution of COP and Robinson’s CQ holds at $\bar{x}$. Then

Proof.

“$T_\mathcal{F}(\bar{x})\subset S(\bar{x})$”. For any $d\in T_\mathcal{F}(\bar{x})$, by the definition of Bouligand cone we know there exists $t^k\downarrow 0$ and $(x^k)\subset\mathcal{F}$ with $x^k = \bar{x}+t^kd + o(t^k)$. Note that

Thus for all $t^k$ we have

”$\text{RCQ}:S(\bar{x})\subset T_\mathcal{F}(\bar{x})$”. For any $d\in S(\bar{x})$ we have $G’(\bar{x})d\in T_\mathcal{K}(G(\bar{x}))$. Then by definition of Bouligand cone there exists $t^k\downarrow0$ such that

Since RCQ holds at $\bar{x}$, there exists a $N>0$ such that for all $k>N$ we have

Corollary 1. (Optimal Solution of LOP) Suppose $\bar{x}$ is a locally optimal solution of COP and Robinson’s CQ holds at $\bar{x}$. Then $d=0$ is an optimal solution of the LOP (the linearization of OP) defined below

11. Proper Lagrange Multiplier Set

Remember the Lagrange multiplier set at $\bar{x}$ is

Here, we say a set is proper if it is nonempty, convex, bounded and closed.

Note. ($\mathcal{M}(\bar{x})$ is closed) For any sequence $(\mu^k)\subset\mathcal{M}(\bar{x})$ with $\mu^k\to\mu$, we must have $-\mu\in\mathcal{N}_\mathcal{K}(G(\bar{x}))$ by the closeness of normal cone. Also since ${\mu:\nabla f(\bar{x})-\nabla G(\bar{x})\mu=0}$ is an affine subspace which is closed, we know that $\mu\in\mathcal{M}(\bar{x})$. END.

Note. ($\mathcal{M}(\bar{x})$ is convex) For any $\mu_1$ and $\mu_2$ in $\mathcal{M}(\bar{x})$ we know for all $t\in[0,1]$ we have

and

Thus, $\mathcal{M}(\bar{x})$ is convex. END.

Note. ($\mathcal{M}(\bar{x})$ is bounded $\Rightarrow$ RCQ holds at $\bar{x}$) If the RCQ does NOT hold at $\bar{x}$, then by Proposition 1 (c) we have

Thus there exists a $\mu_0\not=0$ such that $-\mu_0\in\mathcal{N}_\mathcal{K}(G(\bar{x}))$ and $-\mu_0\in[G’(\bar{x})\mathcal{X}]^\perp$ $\Leftrightarrow$ $\langle\mu_0,G’(\bar{x})x\rangle=0$ for all $x\in\mathcal{X}$ $\Leftrightarrow$ $\nabla G(\bar{x})\mu_0=0$. For any $\mu\in\mathcal{M}(\bar{x})$ we can verify that $\mu+t\mu_0\in\mathcal{M}(\bar{x})$ for all $t>0$: First,

Then, by the closeness and convexity of normal cone $\mathcal{N}_\mathcal{K}(G(\bar{x}))$ we have

Thus, $\mathcal{M}(\bar{x})$ is unbounded. END.

Note. (RCQ holds at $\bar{x}$ $\Rightarrow$ $\mathcal{M}(\bar{x})$ is bounded) If $\mathcal{M}(\bar{x})$ is unbounded then there exists a sequence $(\mu^k)\subset\mathcal{M}(\bar{x})$ with

Then we have

(1) $\mu^k\in-\mathcal{N}_{\mathcal{K}}(G(\bar{x}))$. It implies that

and then $\mu\in-\mathcal{N}_{\mathcal{K}}(G(\bar{x}))$ by the closeness of normal cone.

(2) $\nabla f(\bar{x}) - \nabla G(\bar{x})\mu^k = 0$. Divided by $|\mu^k|$ on both sides and we have

Letting $k\to\infty$ we have $\nabla G(\bar{x})\mu=0$ $\Leftrightarrow$ $\mu\in[G’(\bar{x})\mathcal{X}]^\perp$.

Then (1) and (2) contradict to Proposition 1 (c). END.

Note. (RCQ holds at $\bar{x}\Rightarrow \mathcal{M}(\bar{x})$ is nonempty) Denote by

It’s obvious that $\bar{N}$ is a convex cone. Now we want to show its closeness. For any $(z^k)\subset\bar{N}$ that converges to $z$, there exists a sequence $(\mu^k)$ with $\mu^k\in-\mathcal{N}_{\mathcal{K}}(G(\bar{x}))$ and $z^k=\nabla G(\bar{x})\mu^k$ for all $k$.

Case A, if $(\mu^k)$ is bounded then there must be a sub-sequence $(\mu^{i_k})$ that converges to $\mu\in-\mathcal{N}_\mathcal{K}(\bar{x})$ by the closeness of normal cone. Then we must have $z^{i_k}\to z’$ with $z’=\nabla G(\bar{x})\mu\in\bar{N}$. Since $z^k\to z$ we know $z’=z$. Thus $z\in\bar{N}$.

Case B, if $(\mu^k)$ is unbounded, then by the closeness of normal cone there must be a subsequence

where $\mu\not=0$. Note that $(z^k)$ is bounded since it converges, we know

which implies that $\nabla G(\bar{x})\mu=0$, and furthermore $\mu\in[G’(\bar{x})\mathcal{X}]^\perp$. That contradicts to Robinson’s CQ by Proposition 1 (c). Thus, $\bar{N}$ is closed. Finally, $\bar{N}$ is a convex closed cone.

Next, we shall show that $\nabla f(\bar{x})\in\bar{N}$. If NOT, let

Since $\bar{N}$ is a closed convex cone, by the properties of metric projection we have

and for all $s\in\bar{N}$ we have

It follows that for all $\mu\in-\mathcal{N}_\mathcal{K}(G(\bar{x}))$ we have

meaning that $G’(\bar{x})\bar{d}\in\mathcal{T}_\mathcal{K}(G(\bar{x}))$. Then $\nabla f(\bar{x})\bar{d}<0$ contradicts to Corollary 1 (Since $d=0$ optimizes the LOP, we should have $\nabla f(\bar{x})\bar{d}\ge0$ if $d\not=0$). END.

Finally, we conclude the main result of this note in the following Proposition:

Proposition 8. (RCQ $\Leftrightarrow$ Proper Lagrange Multiplier) Suppose $\bar{x}$ is a locally optimal solution of COP. Then the Lagrange multiplier set $\mathcal{M}(\bar{x})\subset\mathcal{Y}$ is a nonempty, convex, bounded and compact set if and only if Robinson’s CQ holds at $\bar{x}$.

从海浪里，到沙滩上，

处处散发出知了的叫声。

我长梦初醒，

惊觉被丢弃在了密林深处，

只带了一颗打火石和一把指南针。

你是时光的相册，

记录着每一页没有匆匆的遇见。

我寻着记忆，

从北方黑色的森林，

一路追到了南洋荒芜的原野。

你挥毫一笔，

将点缀着晚夏独特忧伤的玫瑰园，

变成了光影里一颗漂亮的珍珠。

我则嬉笑怒骂，

只把散落满地的爆米花，

沉默在了波澜不惊的长河里。

每当我从长夜中惊醒，

就会看见车窗外那座雨后缤纷的彩虹。

你是躲藏起来的黄毛野兔，

还有花蒲里消失不见的西瓜虫。

很多人都告诉我说曾见过你，

见你乘小舟推开满池墨绿的荷叶，

见你用稻花撩逗几只丑陋的青蛙。

我曾故意推倒了一部残局，

让棋子散落成漫天星斗。

即使时常传来欢欣鼓舞的雷声，

骤雨前的狂风也再没有了往日的清凉。

我主动逃到了一层薄雾之后，

回来时已然判若两人。

你还是过去的样子，

虽然越久越散发出香醇。

门庭内外，

我会为你守护那朵尚未衰败的一串红。

我知道，你从来不曾错过，

在晨雾弥漫的山路间，

那人际罕至的板桥霜色。

我只能暗暗窥视，

看着冰凉的晨风摇动枝头的枯叶，

会否落在你曾经的肩头。

向使相隔千年，

你要惩罚痴迷的吊客，

也不必在天涯海角，

留下湘水边寒林秋草的蛛丝马迹。

我只能溯流而上，

粘着白露满身，

不知轻轻划过的苍苍那片，

是斑竹林，还是芦苇丛。

再不想遇见你，

也会在历史深处的角落不期而遇。

我是失意的凡人，

你是洛水的惊鸿。

我知道你从来不曾错过，

每一片绿叶和每一朵红花的尽头。

自从我在梧桐树下，

被那颗冰凉的雨滴打湿的一刻，

你的形象就再也挥之不去，

而我也从未再一次与你相逢。

雨季如约而至。

我又忘了带伞——

Pasir Panjang，这条常走常新的路，

仿佛是在十五年前

千里之外的另一座雷雨交加的公交车站，

那个抱着书包的湿漉漉的小孩子，

闯过了冰凉的鼓点，

踏破了时间的凝滞。

暴雨敲击着路面，

来也匆匆，去也匆匆，

没过那些豪言壮语，

却冲不掉固执的瘢痕，

也挡不住准点的班车。

风也好，雨也罢，

混在机车的轰鸣里，

像戏台上一首迷人的老情歌。

细雨里的班丹水库，

湖面上的微波在积云里飞翔，

给坚持不懈的跑步者

送来强劲的风。

走在潮湿的路面上，

我仿佛能看到千里万里，云海翻腾。

新生从雨中来，

希望自雨中生。

在两侧高楼的阴影里，

杂乱、局促的热带植物，

不顾一切，

从水泥的裂缝里挣扎求生。

野草的所求，

不过一场雨而已，

而在星光点亮之前，

所有的流连者，

都将回乘最后一趟班车。

自己给自己放一天的假，

窝在小房间里读一本书，

小水壶中沏一杯茶。

Work 压不住，

快乐似神仙。

早晨起来已经 9 点半了，

懒洋洋在屋里四处溜达溜达。

胡乱在小面包里抹上几点黄油，

就开始激动地盘算有哪些人间好剧可供消遣。

嘴角露出姨母笑，

手头连上互联网。

不知不觉已到中午，

肚子开始咕咕直叫，

于是正襟危坐，顾望怀愁，

盘算着点什么食物方能效用最大化。

不久电话响了，

只见凌波微步，罗袜生尘。

猛然间想到了昨天的某个步骤似乎有错，

不觉头皮一紧，精移神骇，

慌忙间提笔疯狂 check，

不久便神光离合，忽焉思散。

古人有诗云：

正是人间四月天，

太阳熏地梦沉沉。

遥想关公当年勇，

最后还不是走麦城……

Reference

• Bauschke, H. H., & Combettes, P. L. (2011). Convex analysis and monotone operator theory in Hilbert spaces (Vol. 408). New York: Springer.

• Section 5.1-5.2 of Bauschke and Combettes (2011).

1. Fejér Monotone Sequence

Def. 5.1 (Fejér Monotonicity) Let $C$ be a nonemoty subset of $\mathcal{H}$ and let $(x_n)$ be a sequence in $\mathcal{H}$. Then $(x_n)$ is Fejér monotone w.r.t. $C$ if

for all $x\in C$ and $n\in N$.

Example 5.2 A bounded increasing sequence $(x_n)$ is Fejér Monotone with respect to the set $(\sup x_n, +\infty)$.

Definition (Quasi-nonexpansiveness) For any $\emptyset\not=D\subset\mathcal{H}$, an operator $T:D\to D$ such that $\text{Fix}(T)\not=\emptyset$ is quasi-nonexpansive if

for all $x\in D$ and $p\in\text{Fix}(T)$.

Note. Non-expansiveness implies quasi-non-expansiveness:

for all $p\in\text{Fix}(T)$.

Example 5.3 Let $\emptyset\not=D\subset\mathcal{H}$ and let $T:D\to D$ be a quasi-nonexpansive operator such that $\text{Fix}(D)\not=\emptyset$. Then $(x_n)$ generated by the fixed point iterations of $T$ is Fejér Monotone.

Proposition 5.4 (Fejér monotonicity; General Set) Let $\emptyset\not=C\subset\mathcal{H}$. Suppose $(x_n)\subset\mathcal{H}$ is Fejér monotone with respect to $C$. Then

i) $(x_n)$ is bounded by $B(x,\Vert x_0-x\Vert)$ for any $x\in C$

Proof. By definition 5.1.

ii) For every $x\in C$ we have $(\Vert x_n-x \Vert)$ converges

Proof. By definition 5.1.

iii) $(d_C(x_n))$ is decreasing and converges.

Proof. By contradiction. If $d_C(x_{n+1}) > d_C(x_n)$ for some $n\in N$, then

iv) $\Vert x_{n+m}-x_n \Vert \le 2d_C(x_n)$ for all $m,n\in N$.

Proof. $\Vert x_{n+m}-x_n \Vert \le \Vert x_{n+m}-x \Vert + \Vert x_n-x \Vert \le 2\Vert x_n-x \Vert$ for all $x\in C$.

2. Shadow of Fejér Monotone Sequence

Proposition 5.7 (Shadow Sequence; Fejér monotonicity; Closed Convex Set) Let $\emptyset\not=C\subset\mathcal{H}$ is convex and closed. Suppose $(x_n)\subset\mathcal{H}$ is Fejér monotone with respect to $C$. Then the shadow sequence $(P_C(x_n))$ converges strongly to a point $z\in C$.

Proof. For any $m, n\in N$ we have

The first inequality is because $\Vert P_C(x_n)-x_{n+m}\Vert \le \Vert P_C(x_n)-x_n\Vert$ (Fejér monotonicity). The second inequality is because $\langle P_C(x_n)-P_C(x_{n+m}), x_{n+m}-P_C(x_{n+m}) \rangle \le 0$ (metric projection). By 5.4 (iii) we have $(d_C(x_n))$ decreasing and converge, $(P_C(x_n))$ is a Cauchy sequence.

Proposition 5.9 (Shadow Sequence; Fejér monotonicity; Closed Affine Subspace) Let $C\subset\mathcal{H}$ is a closed affine subspace. Suppose $(x_n)\subset\mathcal{H}$ is Fejér monotone with respect to $C$. Then $P_C(x_n) = P_C(x_0)$ for all $n\in N$.

Proof. For any $n\in N$ and $\alpha \in R$ define $y_\alpha = \alpha P_C(x_0) + (1-\alpha) P_C(x_n) \in C$. Then for all $\alpha \in R$ and $n\in N$ we have

which implies that $y_\alpha - P_C(x_n) \perp x_n - P_C(x_n)$. We have

which implies that $(2\alpha - 1) \Vert P_C(x_n) - P_C(x_0)\Vert^2 \le d_C(x_0)^2$. Letting $\alpha \to +\infty$ and we have $P_C(x_n) = P_C(x_0)$.

3. Convergence of Fejér Monotone Sequence

Proposition 5.10 (Convergence; Fejér monotonicity; Nonempty Interior; Raik) Let $C\subset\mathcal{H}$ such that $\text{int}(C)\not=\emptyset$. Suppose $(x_n)\subset\mathcal{H}$ is Fejér monotone with respect to $C$. Then $(x_n)$ converges strongly and $\sum_n\Vert x_{n+1}-x_n \Vert < +\infty$.

Proof. Pick $x\in \text{int}(C)$ and $\rho > 0$ such that $B(x,\rho) \subset C$. Define a sequence $(z_n)\subset B(x,\rho)$ by

Then $\Vert x_{n+1}-z_n \Vert^2 \le \Vert x_n-z_n \Vert^2$ for all $(x_n)$ that is Fejér monotone with respect to $C$ and for all $n\in N$. We have

Thus

and $(x_n)$ is therefore a Cauchy sequence.

Proposition 5.11 (Convergence; Fejér monotonicity; Closed Convex Set) Let $\emptyset\not=C\subset\mathcal{H}$ is convex and closed. Suppose $(x_n)\subset\mathcal{H}$ is Fejér monotone with respect to $C$. Then the following are equivalent

i) $(x_n)$ converges strongly to a point in $C$.

ii) $(x_n)$ possesses a strong sequential cluster point in $C$.

iii) $\underline{\lim}d_C(x_n) = 0$.

Proof.

i) $\Rightarrow$ ii): Statement ii) is equivalent to say that $(x_n)$ has a sub-sequence that converges to a point in $C$.

ii) $\Rightarrow$ iii): Suppose $x_{k_n}\to x\in C$ then $d_C(x_{k_n}) \le \Vert x_{k_n} - x \Vert \to 0$.

iii) $\Rightarrow$ i): By 5.4 (iii) we know $(d_C(x_n))$ is decreasing and convergent, then $\underline{\lim} d_C(x_n) = 0$ implies that $d_C(x_n)\to 0$. By proposition 5.7 we have $(P_C(x_n))$ converges strongly to some point in $C$, say $P_C(x_n) \to x\in C$, then

Proposition 5.12 (Linear Convergence; Fejér monotonicity; Closed Convex Set) Let $\emptyset\not=C\subset\mathcal{H}$ is convex and closed. Suppose $(x_n)\subset\mathcal{H}$ is Fejér monotone with respect to $C$ and there exists some $\kappa\in[0,1)$ such that for all $n\in N$ we have

Then $(x_n)$ converges linearly to a point $x\in C$; more precisely, for all $n\in N$ we have

Proof. Condition (5.7) implies $\underline{\lim}d_C(x_n) = 0$ which, by Proposition 5.11, implies that $(x_n)$ converges strongly to a point in $C$, say $x_n\to x\in C$. Proposition 5.4 (iv) says for all $m,n\in N$ we have

Let $m\to +\infty$ and we have $\Vert x_n - x \Vert \le 2d_C(x_n) \le 2\kappa^1 d_C(x_{n-1}) \le … \le 2 \kappa^n d_C(x_0)$.

4. Weak Convergence

Definition (Weak Convergence in Hilbert Space) A sequence $(x_n)\subset\mathcal{H}$ is said to converge weakly to a point $x\in \mathcal{H}$ if

The notation $x_n \rightharpoonup x$ is sometimes used to denote this kind of convergence. Referred from Wikipedia.

Corollary 4.18 (Fixed Point; Weak Convergence; Asymptotic regularity) Let $\emptyset\not=D\subset\mathcal{H}$ that is closed and convex and let $T:D\to \mathcal{H}$ be a non-expansive operator. If there exists $(x_n)\subset D$ and $x\in \mathcal{H}$ such that $x_n\rightharpoonup x$ and $x_n-T(x_n)\to 0$, then $x\in\text{Fix}(T)$.

Theorem 5.5 (Weak Convergence; Fejér Monotonicity) Let $\emptyset\not=C\subset\mathcal{H}$ and $(x_n)\subset\mathcal{H}$ is Fejér monotone with respect to $C$. Suppose every weak sequential cluster point of $(x_n)$ belongs to $C$. Then $x_n\rightharpoonup x$.

Proposition 5.13 (Weak Convergence; Asymptotic regularity; Non-expansiveness) Let $\emptyset\not=D\subset\mathcal{H}$ that is closed and convex and let $T:D\to D$ be a non-expansive operator with $\text{Fix}(T)\not=\emptyset$. Let $x_0 \in D$. Set

and suppose that $x_{n}-T(x_n) \to 0$. Then

i) $(x_n)$ converges weakly to a point in $\text{Fix}(T)$.

ii) Suppose that $D = -D$ and that $T$ is odd: $T(-x) = -T(x)$ for all $x\in D$. Then $(x_n)$ converges strongly to a point in $\text{Fix}(T)$.

Proof. By Example 5.3 we have $(x_n)$ is Fejér monotone with respect to $\text{Fix}(T)$.

i) Let $x$ to be a weak cluster point such that $x_{k_n} \rightharpoonup x$. Since $x_{k_n}-T(x_{k_n})\to 0$ we have $x\in \text{Fix}(T)$ by Corollary 4.18. Then by Theorem 5.5 we have $x_n\rightharpoonup \tilde{x} \in \text{Fix}(T)$.

ii) Since $D = -D$ and $D$ is convex we must have $0 = 0.5x + 0.5(-x)\in D$ for all $x\in D$. Since $T$ is odd we must have $T(0) = T(-0) = -T(0) \Rightarrow T(0) = 0$. Thus, $0\in\text{Fix}(T)$. Thus, by Fejér monotonicity we have $\Vert x_{n+1} \Vert \le \Vert x_n \Vert$. Thus, $(\Vert x_n \Vert)$ is decreasing and converges to some point $\ell \ge 0$. For any $m,n\in N$ we have

Thus, $(\Vert x_{n+m}+x_n \Vert)_{n\in N}$ is decreasing for all given $m\in N$. Also we have the identity

Since $T(x_n)-x_n\to 0$ we have $\lim_{n}\Vert x_{n+m}-x_n \Vert=0$. Thus, $\Vert x_{n+m}+x_n \Vert^2 \downarrow 2(\ell^2+\Vert x_m \Vert^2)$ as $n\to+\infty$ for any given $m\in N$. Thus, $\Vert x_{n+m}-x_n \Vert^2\to 4\ell^2-4\ell^2 = 0$ as $m,n\to+\infty$. Thus, $(x_n)$ is a Cauchy sequence and $x_n\to x\in\mathcal{H}$. Since $x\leftarrow x_{n+1}=T(x_n)\to T(x)$ we have $x\in\text{Fix}(T)$.

5. Krasnosel’skii–Mann (KM) Iteration

Corollary (Identity) For any $x, y\in\mathcal{H}$ and $\alpha\in R$ it is easy to verify that

Proposition 5.14 (Weak Convergence; KM Algorithm) Let $\emptyset\not=D\subset\mathcal{H}$ is convex and closed and let $T:D\to D$ be a non-expansive operator with $\text{Fix}(T)\not=\emptyset$. Let $(\lambda_n)\subset[0,1]$ with $\sum_n\lambda_n(1-\lambda_n) = +\infty$. Let $x_0\in D$. Set

Then the following holds:

i) $(x_n)$ is Fejér monotone w.r.t. $\text{Fix}(T)$.

ii) $(Tx_n - x_n)$ converges strongly to zero.

iii) $(x_n)$ converges weakly to a point in $\text{Fix}(T)$.

Proof. Since $x_0\in D$ and $D$ is convex, (5.12) generates a well-defined sequence in $D$.

i) For any $y\in \text{Fix}(T)$ and $n\in N$ we have

The inequality is by the non-expansiveness of $T$.

ii) From (5.13) we have $\lambda_n(1-\lambda_n)\Vert Tx_n-x_n \Vert^2 \le \Vert x_n-y \Vert^2 - \Vert x_{n+1}-y \Vert^2$. Then

Since $\sum_n\lambda_n(1-\lambda_n) = +\infty$ we must have $\underline{\lim}\Vert Tx_n-x_n \Vert=0$. And

Consequently $(Tx_n-x_n)\downarrow 0$.

iii) For any $x\in D$ such that $x_{k_n}\rightharpoonup x$ we have $x\in \text{Fix}(T)$ by Corollary 4.18. Moreover $(x_n)$ is Fejér monotone w.r.t. $\text{Fix}(T)$. Thus, by Proposition 5.5 we have $x_n\rightharpoonup x$.

6. $\alpha$-Averaged Operator

Definition 4.23 ($\alpha$-Averaged Operator) Let $\emptyset\not=D\subset\mathcal{H}$ and let $T:D\to\mathcal{H}$. Let $\alpha\in(0,1)$. Then $T$ is called $\alpha$-averaged if there exists a non-expansive operator $R:D\to\mathcal{H}$ such that $T = (1-\alpha)\text{Id}+\alpha R$.

Proposition 4.24 (Properties of $\alpha$-Averaged Operator - 1) Let $\emptyset\not=D\subset\mathcal{H}$ and $T:D\to\mathcal{H}$. Let $\alpha\in(0,1)$. Then

i) If $T$ is $\alpha$-averaged then it is non-expansive.

ii) If $T$ is non-expansive it is NOT necessarily averaged.

iii) $T$ is firmly non-expansive iff it is $1/2$-averaged.

Proposition 4.25 (Properties of $\alpha$-Averaged Operator - 2) Let $\emptyset\not=D\subset\mathcal{H}$ and $T:D\to\mathcal{H}$. Let $\alpha\in(0,1)$. Then the following are equivalent

i) $T$ is $\alpha$-averaged.

ii) $R\equiv(1-1/\alpha)\text{Id}+(1/\alpha)T$ is non-expansive.

iii) $\Vert T(x)-T(y) \Vert^2 \le \Vert x-y \Vert^2 - \frac{1-\alpha}{\alpha} \Vert(\text{Id}-T)x-(\text{Id}-T)y \Vert^2$ for all $x, y\in D$.

iv) $\Vert T(x) - T(y)\Vert^2 + (1-2\alpha)\Vert x-y\Vert^2 \le 2(1-\alpha)\langle x-y,T(x) - T(y)\rangle$ for all $x,y\in D$.

Proposition 5.15 (Weak Convergence; $\alpha$-Averaged Operator) Let $\alpha\in[0,1]$ and $T:\mathcal{H}\to\mathcal{H}$ be an $\alpha$-averaged operator such that $\text{Fix}(T)\not=\emptyset$. Let $(\lambda_n)$ be a sequence in $[0,1/\alpha]$ with $\sum_n\lambda_n(1-\alpha\lambda_n)= +\infty$. Let $x_0\in\mathcal{H}$ and set

Then the following hold:

i) $(x_n)$ is Fejér monotone w.r.t. $\text{Fix}(T)$.

ii) $(Tx_n - x_n)$ converges strongly to zero.

iii) $(x_n)$ converges weakly to a point in $\text{Fix}(T)$.

Proof.

Set $R = (1-1/\alpha)\text{Id}+1/\alpha T$ for all $n\in N$. Then $\text{Fix}(R) = \text{Fix}(T)$ by definition and $R$ is non-expansive by Proposition 4.25. Then (5.15) is

Since $\alpha\lambda_n\in[0,1]$ and $\sum_n\alpha\lambda_n(1-\alpha\lambda_n) = +\infty$, the result follows Proposition 5.14.

Corollary 5.16 (Special Case: $1/2$-averaged) Let $T:\mathcal{H}\to\mathcal{H}$ be a firmly non-expansive operator with $\text{Fix}(T)\not=\emptyset$. Let $(\lambda_n)$ be a sequence in $[0,2]$ with $\sum_n\lambda_n(2-\lambda_n) = +\infty$ and let $x_0\in\mathcal{H}$. Set $x_{n+1} = x_n +\lambda_n(Tx_n-x_n)$. Then

i) $(x_n)$ is Fejér monotone w.r.t. $\text{Fix}(T)$.

ii) $(Tx_n - x_n)$ converges strongly to zero.

iii) $(x_n)$ converges weakly to a point in $\text{Fix}(T)$.

Proof. When $\alpha = 1/2$ and apply Proposition 4.24 (iii) and Proposition 5.15.

Example 5.17 (Special Case: $1/2$-averaged when $\lambda_n=1$) Let $T:\mathcal{H}\to\mathcal{H}$ be a firmly non-expansive operator such that $\text{Fix}(T)\not=\emptyset$. Let $x_0\in\mathcal{H}$. Set $x_{n+1}=T(x_n)$. Then $(x_n)$ converges weakly to a point in $\text{Fix}(T)$.