Convergence analysis of cautious control

328Science in China: Series F Information Sciences 2006 Vol. 49 No.3 328- 338DOI: 10.1007/s11432-006-0328-zConvergence analysis of cautiouscontrolZHANG Yanxial' & GUO Lei21. Department of Mathematics, Beiing Institute of Technology, Beiing 100081, China;2. Academy of Mathematics and Systems Science, Chinese Academy of Sciences, Bejing 100080, ChinaCorrespondence should be addressed to Guo Lei (email: Lguo@ amss.ac cn)Received January 30, 2005; accepted December 28, 2005AbstractIn this paper, we present a theoretical analysis on stability and convergenceof the cautious control, which has advantages over the traditional certainty equivalenceadaptive control, since it takes the parameter estimation error into account in the design,and is also one-step-ahead optimal in the mean square sense under Gaussian assump-tions.Keywords: adaplive control, cautious control, stability, convergence, leas1-squares, Kalman filter, certaintyequivalence.1 IntroductionOver the past three decades, there have been extensive efforts devoted to adaptive con-trol of linear stochastic systems, and much progress has been made in both theory andapplications (cf. refs. [1-8]).However, most of the stochastic adaptive controllers studied so far are designed mainlybased on the so-called“certainty equivalence principle". This principle means that in thedesign of adaptive controller, the unknown parameter is simply replaced by its on-lineestimate, without taking the parameter estimation error into account. As is well-known,this kind of controller hardly has any optimality at any finite step, and may result in verypoor transient performances of the underlying adaptive control systems.In this paper, we take another view on the design of adaptive control, and investigate theso-called cautious control which may be derived from the standpoint of partially observedstochastic contro] systems (see ref. [3]).Let us consider the following SISO Iinear disc中国煤化工:A(z)yt= B(z)ut- 1MHCNMHG(1.1)where {ye},{ut}， and {wr} are the system output, input and noise processeswww.scichina.com www.springerlink.comConvergence analysis of cautious controlrespectively. We assume thatyt= ut= wt=0, Vt < 0, A(x), B(z2) are polynomials inthe backward-shift operator z,A(z)= 1+a12+... + apP,p≥1,B(z)=b1 +b2z+...+bqz9-1，q≥ 1,wherea;, 1≤i≤p;bj, 1≤j≤q are unknown cofficients; p, q are the upper boundsfor the true orders. Now, let the unknown parameter be denoted asθ={-a1 ... -ap by ... bq]”,(1.2)and introduce the corresponding regressorPt=[yt ... 9t-p+1 u ... Ut-g+lT.(1.3)Then system (1.1) can be rewritten asYh+1=θ"φt +W1+1, t≥0.(1.4)Our control objective is, at any instant t, to construct a feedback control ut based onthe past measurements {yo, .,t,，uo,...,Ut-1} so that the following one- step aheadtracking error is minimized:It≌E(yt+1- +1)尸,(1.5)where {y; } is a known deterministic reference signal.If the parameters of system (1.1) were known, it would be easy to determine the optimalcontrol law as follows:0'pt= tr.From this, ut can be expressed explicitly as46+1- o"中。(1.6) .wheree= [yt ... Yt-p+1) Mu-1... U-g+1IT.In the case where the parameter is unknown, the traditiona]“certainty equivalence prin-ciple" suggests that the unknown parameter θ be replaced by its online estimate 0t, andthus we get the familiar adaptive control law as follows:Uht =在+1-旺西(1.7)bitwhere brt is the estimate of by. Note, however, thatθ=θ+ 0t,θt=θ-0.Hence, obviously, the above controller does not take the parameter estimation error 0tinto account, and so it is hardly to be an optimal one at any time t AIsn 2%t may not be中国煤化工well defined, since the set {brt = 0} may have aTo obtain an optimal controller, let us assumqCNMHGthenoisese-quence {wt} are jointly Gaussian. Assume further that { wt} is a zero mean white noisewith variance o2 Then by the Kalman filter theory (cf. ref. [5]), we have for anyut∈Jr,θt = E01D2i]，P{= ElO,OT|D], .θ=θ- 0r,330Science in China: Series F Information Scienceswhere Jt is the σ-algebra generated by {Yo, ... , yt}, and 0t is the estimate of 0 which maybe generated by the least square algorithm, Pt is the corresponding estimation covariance.From the facts that E[0|Di] = 0 and {wt} is Gaussian and white, we can see thatE[(9+1 - 4+1)91;]= E(0?$Pr-y+1 + TPt + Wt+1)2)|D2]= (OTpe-y*+1)2 +p{P:$t +σ= ("Pe + brtut-y+1)2 +的PrPt + uPb,(t) + 2uP(t)Pr + (1.8)where P(t)= TPr, Pbr(t)= TPl andl is the (p + 1)th column of the d X d identitymatrix withd= p+q.Now, minimizing (1.8) with respect to Ut gives the one-step- ahead optimal controller[b1t(旺中t- yi+1) + P(t)4;]Ut=-(1.9)bit + Pbr (t)The above controller differs from (1.7), because the parameter estimate error measured byPi has been taken into account. Futhermore, it is cautious since the presence of Pbr (t)in the denominator will reduce the magnitude of Ut, once the estimate uncertainty in brtmeasured by Pbr (t) is large (see ref. [3]).2 Main resultsAlthough the Gaussian assumption is used in the derivation of the cautious control inthe last section, the main theorem together with the stability analysis to be given belowdoes indeed not require this assumption.In fact, throughout the sequel, we only need the following standard conditions:(A.1) The noise sequence {Wt,Fi} is a martingale difference sequence i.e.,E[W+1|Ft] = 0, and satisfiesE [w2+1F]=o2>0 a.s.(2.1)sup E [|wt+1|P\Ft] <∞a.s. for someβ> 2.(2.2)(A.2)B(z)≠0, Vz with |2|≤1 (the minimum phase condition).(A.3) {yt } is a bounded reference sequence independent of {wt}. .We remark that if {dt } is a nondecreasing positive deterministic sequence such thatwi2 = O(d) a.s.(2.3)then under condition (A.1), dt can be taken asd=t，如r∈(21),(2.4)中国煤化工where β is given by (A.1) (cf. ref. [7]).CHCNM HG_Instead of the one-step-ahead tracking eror It, wc Uullsiuci wic luiuwing averagedtracking errorJt江(y;-y")*.Convergence analysis of cautious control331DefineR≌> (yi-y:- w1)*.(2.5)Then by the conditin (A.1), we know that for any adaptive sequence {u, F}, the asymp-totic lower bound to Jt is o2, andRr=o(t)台lim Ji=σ2,where σ2 is defined by (2.1). Furthemmore, once R = o(t) is proved, the global stability,ie.2j : ,(号+号) = O(t), can be derived easily by the conditions (A.1)-(A.3).Now we introduce the standard least-squares (LS) algorithm for the estimation of theunknown parameter θ :0r+1 = 0t + arPt4e(yt+1 - sI 0),(2.6)P+1= P:- atPrsrp2 P,(2.7)ar=(1 + oI Pt4)-',(2.8)where the initial values 0o, P > 0 can be chosen arbitrarily.Let brt be the estimate for b1 given by Ot, and denoteD≌{brt≠0, Vt; liminf√log(t +rt-1)brt|≠0},(2.9)wherer:=1+Iloll2.(2.10)=0The main result of this paper is as follows.Theorem 2.1. For system (1.1), let the conditions (A.1)-(A.3) be satisfied, and let thecontrol law be defined by (1.9). Then the closed-loop system is globally stable, optimal(in the sense that Rt = o()), and has the following rate of convergence on set D:Ri = O(logt +Et),(2.11)εt = (logt) max{8j"d}, Vε> 0,(2.12)δ≌tr(Pr- P+1). .(2.13)3 Proofs of the main results中国煤化工To prove Theorem 2.1, we need to present sevIntroduce the following notation:MYHCNMHGA_(Tp)21+stPrsr_We then have332Science in China: Series F In formation SciencesLemma 3.18. Under conditions (A.1) and (A.2), for any initial values (Oo, Po), if{un} is adaptedtoGr兰o(yj, y*+1; j≤t), then the estimate {0x} given by the LS -basedalgorithm (2.6)- (2.8) satisfies(H.1)|0lIP = O(logr -1) a.s.(H.2)2Q; = O(logrt) a.s.i=1where ri is defined by (2.10).For simplicity, we will sometimes omit the phrase“a.s. on D”in the sequel, and allrelationships should be understood to be held on D with a possible exceptional set ofprobability zero.Lemma 3.2.Consider the closed-loop system (1.1) with the control law given by(1.9). If conditions (A. l)-(A.3) are satisfied, then there exists a positive random process{Lr} such thaty≤Lt， Lt+1≤(入+cft)Lt+ξt a.s. on D,where the constantsλ∈(0,1), c> 0, andft= [ar0t log(t + r)]2 + ar[8t log(t + r)]*+ ar8+ [8,log(t +r)]4 + 82 log(t + r),(3.1) .ξt = O(d,log*(t + r)).(3.2)Proof. First of all, from (1 .9), we know thatbrt(b2eUMu +瞪q) = -(20 (t)urt + P{t)Pt) + 1t4+1,that isb1t(哩4t- yi+1)= - P(t)Pt.(3.3)Next, we give an estimation of |P(t)4t|2. From the LS algorithm(2.6)- (2.8), we have|P(t)pe|2 = lI PrPrpt Pl= [r(P- P+1)iht=(1 + ρt Pp4)"(P- P+1)l≤[1 +4?(Pi- Pr+1)4t + pI Pr+14p}8t≤8(2 + 5l),(3.4) .where we have used the fact thatpf Pr+14t≤1.From the definition of D, we know that there exists a random variable M such that1bp≤Mlog(t+rt-)(3.5)中国煤化工Then combining (3.3), (3.4) and (3.5), we have :TYHCNMH G(0FPt- y+1)°≤δ(2 + 8lpll)M log(t +rt-1) a.s. onD. (3.6)Now, by (1.4), we haveYe+1=θT4r +W+1=θ"ψt + ('Pt - yi+x)+ t+1 + W+1.(3.7)Corvergence analysis of cautious control333From the definition of at, we know that(Tp)2 =ax(1 + p? Prser)=Qe(1 + φE(P - Pr+1)Pe + o? Pt+rpe)≤ar(2 + 8llylI|2).(3.8)From (3.6) and the above, we know that on set D,y崛+1≤3(OTsot)2 + 3(0"pt-yi+1)2 + 3(y;+1 + wW+1)2≤3ar(2 + sl!l!?!)) + 3M8(2 + 8p!e!)log(t +r-1) + O()= (3at + 3Mδ log(t + r1))lol|)+(6at + 6Mδ% log(t +r-1)) + O(d).(3.9)By Lemma 3.1, we know that Qt = O(log ri). Also, from> δ;=> (trP;-trPj+1)≤trP≤∞,j=0we know that δt - + 0. Therefore,y41≤(3xe + 3Mδ& log(t + r1))lloll2 + O(log(t + r)) + O(d). (3.10)By the stability of B(z), it is seen that there exists λ∈(0, 1) such that心1=o(2>-h)+ O(dt).(3.11)i=0Hencep-1q-1tlorll-un =师+二呢。=o(二x-?) + 0(d).(3.12)i=lBy the property of (H.1), (3.5) and the fact that P(t) is bounded, it follows from (1.9) thatu2 = (og(+r-)(三号+己吗)+log(t +rt 1)).(3.13)i=1Moreover, putting (3.11) into (3.13), we have(3.14)吗= o(log?(t +r-)l(Ex ~y2 +d)).Hence, by (3.12) and the above equality, we haveI1ol = O(L1log2(t + Tt-1)) + O(dr log2(t +-1)),(3.15)whereLt兰2t=0 λt-y?. Note that中国煤化工brue=旺pPr + (Of'4i- yi+1)YHCNMHGby (3.12), we haveb经≤4(6T42)2+ 4(0"4o- Vi+)2 + 4(y+1)2 + 4(rue- 0T"p,)2= 4("p)2 + 4(0Zρ:- 4i+1)2 + O(Lr + du).334Science in China: Series F Information SciencesSimilar to the proof of (3.10), it is known thatuq = O(at + δt log(t + r- 1)]8l2)+ O(Lt) + O(d + log(t +r:)). (3.16)From (3.12) and the above, we can getloll = O([cx + δe log(t +rt- 1i412)+) O(Le) + O(dn + log(t + ro)).Substituting (3.15) into this, we haveIlol = O([at + 8log(t +r-1){8.Lt log2(t + rt-1))+O([at + δt log(t + Tt- n)]8rd log"(t +rt-1))+O(Lt +d + log(t + rt))= O([at + δt log(t + rt-)]8rLflog2(t + Tt-1))+0(Lt) + O(de log*(t +r:)).(3.17)Finally, putting the above into (3. 10), we have呢41= O(|at + δt log(t + rt- 1)]82 log2(t + rt-1))L+O([at + δt log(t + rt-1)]8x)Lt+O([at + δt log(t + rt-1)]8tdt log*(t+ rt))+O(d + log(t + ri))= O([arδt log(t +r)2 +a[δe log(t+rt)]3 + arδt+[8e log(t + r)]4 + 8? log(t + r))Lt+O(dt log*(t + r()).(3.18)That is to say, there exists a random variablec > 0 such thaty+1≤cfiLt + ξt,where ft, ξt are defined in (3.1) and (3.2) respectively. By the definition of Lt, we knowthaty?≤Lt. Furthermore,Lt+1=yt+1 +λLt≤(入+ cft)Lt+ ξt-Hence the lemma is proved.Lemma 3.3. Under the conditions of Lemma 3.2, we havelHIl= O((t+rt)=d) a.s.on D，Vε> 0.Proof. By Lemma 3.2, we know thatLt+1≤x+[II(1+>-4ef)[Lo+Zx-if I (1 +λ-'cf;)]s.(3.19) .i=0中国煤化工We proceed to estimate the product Iij=i+1(1MYHCNMH GFirst, by Lemma 3.1, for anyε > 0, there existsd > 0 such thatδ2 aj≤εlog(r)， Vt.j=0Convergence analysis of cautious control335Also, since 2j=oδj <∞, there is an integer io > 0 suficiently large such that1/2 o()“2oδj≤ε， Vi≥io.j=iWithout Ioss of generality, we assume that0 < δ < 1 and f > 1. Note that1+λ-1cfj=1+>-'((azδj log(j +r,)2 + as[8; log(j + r,)}8+a;8j + [8; log(i +r;)]* + 8f}log(j +r;))≤(1+ ~-'c[axzδ; log(j + rj)]2)(1 + )-ca,[8; log(j十r;)]3)x(1 + λ 'ca;8)(1 +入-1c[8; log(j +r,)]4)x(1 + )-'c63 log(j +ry)).(3.20)As proved in ref. [6],I (1 + λ-'c[as8, log(j +r:)1)≤(t+r)*,(3.21)j=i+1I (1 +λ-ca;8j)= O(rf)(3.22)j=i+1.Then by the inequalities1+xy≤(1 +x)(1+y)， x≥0, y≥0,1 +xn≤erx， n≥1, x≥0,we know that, forallt≥i≥io,II (1 +λ-1c[8; log(j + r,)4)≤exp(号)" 亡6log(i+r)≤ep(4(9)"二 s(]xg(t+r)≤(t +r)*,(3.23)II (1 +>-ca}{6; log(j +r)9)t≤II(+8a) II (1+ 510og"(i+r)≤exp(δ 2 aj)exp中国煤化工源IYHCNMHG≤rexp((0()” 26j]log(t+r)3= 0((t +rt)"*),(3.24)336Science in China: Series F Information SciencesandII (1 +λ- 'c82 log(j +r))j=i+1≤II (1+>-'c8j) II (1 +δ; log(j +r;))≤exp.(∈亡s)exp((亡δ; log(i +r;)= o( exp (og(+r)l E 6,))= 0(t+r).(3.25)Then combining (3.20)- -(3.25), we haveI (1 +>-'cf;)=O((t +r)"*)， i≥io.(3.26)Finally, substituting this into (3.19), and note that0<入< 1, we have .L+1= O((t +r)redu log*(t +r))，Ve>0.Then by Lemma 3.2 and the arbitrariness of E, we know thaty{+1≤Lt+1=(t+rx)"d)，Ve>0.From this and (3.11), we know thatu? = O((t + r)*d), Vε > 0. Finally, from (3.12),we can see the lemma is true.Now we are in the place to prove the main result.Proof of Theorem 2.1. By (3.7), we haveRt+1=> (yj+1-5+1 - Wj+1)P=2(的s + (q,- y+1)2≤2> (时4;)*+222 (0p3-y5+1)9.(3.27)j=0From (3.6) and (3.8), it follows thatR+1≤2Z a;(2 + 81l2) + 2M 22 [8,(2 + 8l|)log(j +r;- 1)]中国煤化工=45 ,a;+4M〉δj log(j +r;YHCNMH G= =0+0(二lc[a;δ; +号}log(j +r;-1)(G + r,)"d,). j=0Convergence analysis of cautious control337≤O(logrt) + 4Mlog(t+rt-1)88j+o( as{(j+rj)}d}{2 ay+ Z8 lg(j+r-1)])j=0= o(log(t +r:)) + o( max{8j(Gj + r;)°d}log(t +r)).(3.28)Therefore, for (2.11), it sufces to prove that rt = O(t). From the above, we haveRt+1 = O((t + rt)*dt).By (3.27) and the assumptions on {u; } and {w; }, it follows that二明}= o(Re+1) + O(t)=O((t +rt)d) +O(t)，Vε> 0.(3.29)From this and condition (A.2), it follows that二吗= o(二)+0(二吲)= 0((t+r)*d)+ O(t)， Vε> 0.(3.30)Then from the definition of rt, we have for anyε > 0,rt=1+ > l41|"= O((t + r)*d) + O(t)=O{t)+O((t+r)*t7)，Vγ∈ (2/B,1).By taking r small enough such thatε +γ< 1, we getr't.t=(1)+(()0)=0()+0((1+)From this we getr = O(t), and henceRl+i = O(logt) +O(ex) a.s. onD,where Et is defined by (2.12). Obviously, Rt = o(t). Hence the proof of the theorem iscompleted.Remark 3.1.In the above, we have discussed the SISO system. Actually the resultin Theorem 2.1 can be extended 10 the multidimensional case. That is, if {ut}, {u},{0r}are all m-dimensional in (1.1), andA(z)=1+A1z+..+ Ap中国煤化工B(z)= B1+ B2z+...+.:MYHCNMHGwhere Ai, 1≤i≤p; Bj, 1≤j≤q are unknown matrix cofficients and p, q are theupper bounds for the tnue orders, we can also obtain the cautious control under the corre-sponding assumptions (see ref. [7]), which has the same convergence rate as in Theorem.338Science in China: Series F Information Sciences2.1. The analysis is similar to the SISO case, but more complicated.AcknowledgementsThis work was supported by the National Natural Science Foundation ofChina (Grant Nos. 60221301 and 60334040), and the Basic Research Funds of Beijing Institute ofTechnology.References1 Goodwin G C. Sin, K S. Adaptive Filtering, Prediction and Control. Eng]ewood Cliffs. NJ: Prentice Hall, 19842 Cains P E. Linear Stochastic Systems. New York: Wiley, 1988Astr'm K J, Wittenmark B. Adaptive Control. Reading. MA: Addison-Wesley, 19954 Kumar P R. Varaiya P. Stochastic Systems: Estimation, Identifcation and Adaptive Control. Englewood Ciffs,NJ: Prentice Hall, 19865 Chen H F, Guo L. Identifcation and Stochastic Adaptive Control. Boston: Birkhauser, 19916 Guo L. Convergence and loganithm laws of self- tuning regulators. Automatica, 1995, 31(3): 435- 4507 Guo L, Chen H F. The Astrom-Wittenmark self- tuning regulator revisited and EL .S-based adaptive trackers. IEEETrans Automat Contr, 1991, 36(7): 802-8128 Guo L. Self- convergence of weighted least-squares with applications to stochastic adaptive control. IEEE TransAutomat Coatr, 1996, 41(1): 79- -89中国煤化工MHCNM HG