From efbeca3c322fa62a95b51ebc5670a6d446dcdebc Mon Sep 17 00:00:00 2001 From: wakonig_k Date: Tue, 23 Jul 2024 18:02:46 +0200 Subject: [PATCH] docs(messaging): added first draft of bec messaging docs --- docs/source/assets/messaging_system.drawio | 40 +++++++++ docs/source/assets/messaging_system.png | Bin 0 -> 12575 bytes .../developer/data_acess/bec_messaging.md | 85 ++++++++++++++++++ 3 files changed, 125 insertions(+) create mode 100644 docs/source/assets/messaging_system.drawio create mode 100644 docs/source/assets/messaging_system.png create mode 100644 docs/source/developer/data_acess/bec_messaging.md diff --git a/docs/source/assets/messaging_system.drawio b/docs/source/assets/messaging_system.drawio new file mode 100644 index 00000000..c6a679c2 --- /dev/null +++ b/docs/source/assets/messaging_system.drawio @@ -0,0 +1,40 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docs/source/assets/messaging_system.png b/docs/source/assets/messaging_system.png new file mode 100644 index 0000000000000000000000000000000000000000..e4733e3b94e376372abfa83f8a6224f36bcdf489 GIT binary patch literal 12575 zcmd^m2{_bk+qb0%iAV{BB1;KlFZ(`rAz32JU<$*`*v(iXB}EfSwqztqq=f7#OV&n2 zSqjmpEE(G{yw_OnyZe5g_j#Z1`=0kW?&tZAgPH&Tyw2;qu5-VxpTiY16FpXD9%dRE z8dj*j_9+?~S`2u;xPt+FI^AlC17EbLQ+k>-kK6f(G&FmlXq_`?uRu4rCyYi|Mr-3q z7$W6?K%s?Yw1puM9Z#4G8j0`*kAUulaD}_UVXhlv5GjbPgp`5=M8;BDPFO}mmii&3 za9m1B!FGe+$sOjsDaaTR4EOYO5{Br?9hU;4M2($X;NEBi%0pO26TCyc(J)W&4anfr z)B=2*0l(5xj?!|D3S!_i;~6mT4l0PGOigaK#-odqp_u@iNBAT3rhEtzx8QP^9{}YVMr7lm~GvNTZ>aC zql0{48&p@AKimcEgi1xZIC*Z8r8g+f2&5|vxxoqS1re6f5th;btkfSFEmydcJJQJu zym&hSb=H?L1Ll_4oDSdgQ6L1R>8)?(3)l99YinEDAS@uLKTF$^=@qDlbn@{qM!2qf z!ZmP1EHD(Ns&Y}j$jnn(l!$os`jel}%>WNL8fM|+1fpI>I{?HQpn0IZJi(jv zdKI9~2AP9~ZH3;JgMQa&UCDoJ*58e*2Lri|Mg{?~n-XPYH%+n`%F;@k?*YFDhBQ?G zbt@jf1cm%2&1o|h-G38J<@{&-1$Z{y{g1a#t;2tciofd$u4Y zE<$~_p*|l6=gsvuZEA7f8q%gl^7@N6sQl}bHW;a|)Opla0~#bU==W*sZ(H*=whD9o zn={3qnh-d(A^ec*y$nqGyCr(FBcq*=?y$|n#oy&yFXjKZEBxK2_Di*t+YFvxi{%y# z3G;M9!~OqwM*4dd^q<+n?;A0rw*B8*>z`{nB({`)1jmZ16$ zIN`S~Y$>;4`TvV0Wx#a;wCmpv5dYXRo80SN`+ws6=bF3XMil-pWYBL}L(PTXlIGt% zgVwv)KQ{gEQS_TvAyR*-43gPO*sXAu*-YxKaQU}CG|{SE>B8Vu{BTcFd% zz~%%hc({R8pr?(_(a`WkLA5n3108;(GnJWiFEs8m7mtojOzafUT02F!%%GB|4?iTF za|+(c2Rom^$z%3bNF1)dgEiJtTP8`gEypIireiW=IppkA`9j)q25Jy9Uok%$Xg@XJ z&i1XMC!l~bF#du|+cn9<@ZomxSCu|<*KV`&jZ9DQO7keRX^n6{JY7KOHQ~L?C4T$% zb(NxS1;W_=LleBj`eS)qBfJJ=sLjdYAlahB@(;A?^AL*L++Nw46XJXCLU-jxq~lAv zY1QUh(g2vOec;`xous2y1`px!A*b1@o%9AjwU8ZVD?;S#nud9;3cA~T?;SB!o z_WHRi&s|$_QNpeS!h0U2$a-`NtX*wB6M@6tkY1%MI0h|!NrsgU_}^re@2%a3&&|Zz zwx66*Uiz9H|8p0f!GBP&$e(tCcPZx5@KeN4_31sAQ+kRnk@7MmE2lM=K3!xR1;4lK z5LH(4189r#<;jXKvmI*5rm9QY_l;5xwWTT~n`i4n+wRDH`B6G_3)P`wt-AbOMrpEa z#Pw5c1SD)VbZP!0;>!)agVIUMx20WNbJP8C6_YOv{d=7lm&P;0Mn7CAlp9$EkAxh< zedXkdhdIu#W09R2jLr3IYR6iVC6iMT)iyi|-pA@=*ph8a&Np{vX~AaO6q4!Ll+9FE z7hRMl9%PR`X*}5S&^<@4=m{l;A1gW(I(x14wzSrDr7`Y4u&o@^Ox1>r(thMszTmG} zW8apiYK9hHO4h6#5=rxkwD*0@rTa2FmPa|TsB)%(7x`+pUQ>dgR@BXmp>ag@(tUeW zo7|XC8PLB)INAOX_%2KtxgwGwrCPK~{?-E26GM%)-q7R6JbSlfNWYqwn}KUNUSV`g z!tMxIs>bHyX*(}i=mbz$UKQwA_(m{>&w0M~MYcD5n$>wV|KVeZw2Jeq9T!=)cYjHC z^fu`qPeYkJF2A8M9xapoy6qSCy_ z#{B>Yo`BXgN9?3^}Em6yX071$|o;)R%+b}+?k>wMy(r!?CtbIoFA zWwiNbn=%OARHHkxQ2znH_x7HdvrpkWzI&#Uq_%5C4nONMk-5V4(Iovux1y4Wm}mdv zvnRFUuJOp>Zknq0ont;CM0w)UXaER1GepwSxVjS`$E!jX$Ygor;x359ZT4^MN2LdmaI$H z)|O^THmmh~_Ez`j%4|nVneRw3c^ei4dx-wrME4iq40#OHLr6xoCC5gC9B!n0_60g` z>6#k1XXAl@M9Jz!&B@AH6AM-xjfW*|0;J(1cKL^KiGC3-iT2n{5^Y%3T#wB4r*l1a zErm6!=jr%IIOAl+cXBYJZV6!^KHd3N*JRAEX>Jpr8XfjtQ(+;FylR5WnwV{4$h0Uw z{KS{};!b^rbJ`UPgCWKxTF$+0sa{6sZWh(7t%hj5vz+KtjX5V69Lc_RN6IM@{o_No zvX-EF_d=^o$8Du)trglwOsk%=BS*2UYp)#hM&VO6Yt_&*i^rlFL{6_HTK2?ncOg(L^Qv$&s&{+FbW@^fXNQ7Xc|K9 z2Y2sS1ItK|EAK7mNfD&F76cX-;nv%^m z4~~u^r+Op%QNkkujQn+TgXhsK`|cR&L&~PAms?Ajg0BzJ=*Z90vG-kasnH{E>jh;I zmNCP3S>5yF^8)4RT6#j37UQf+Q%r082n?qxLSrKE%-SJHXMQjQ;SwlJvG4y8hBuuS z9Ox8)<#XptNR9tw(LCbupcTT)#Svx1j_kkxnMGecjl7~Naz+3p-b|kR#(U+ulA_!x z*-zH8;>ZS;-j**GTU~g?q;YV)`a$Q+z!T1hO#EH%7!!0w;J@E zCfo_CgvUcG#&=|Aq9tyUY}g>Xi@9<+Vxh6OGE_p`=9rl2=Bw;HCmN&Mgred|T7EH>+9MhFLl!14kpxFQXR|ey<=0$q4_jno;!bjEktmhLuXSp%Ewy*#)fOgIHVwFEV&dXQ;?s^T zd_<^ek^4d;@ll5&YC@W?V4k>*Hre0d!gp+!;2jlZe2aRb=^xFlSPeUjnHs*t!V$Sg zOOr378;i{mC2;9bZ~_AJ z<*4yH2F#zNcCaqN>LR!6vI!jy-d5*(5SjbNMHwPOLrB)e34TNN{~YXcEOu??X}@hl zq~EJB&3DwW%O6E6{FT}d?&9MuV@7Y;?6Z=^4w+!3AZ87U^8w zd!)ASs>uEHxKN1LJ#jCT2!HiXGa)KXkV3BvySXbySt*ATPtMVDPT1jl;914 zIh95D>Mvg1Cn+x(M>UOi|CE4?%&q`NA4USdj%a3=!7LMVx6u_wHR1hVK0Fe{$iK`v zR@Y!&X^NgU7x6hpgut)?FePsCwG>K^*eg_)~oz(1zZvG z#JEo}!Bk|=XX}E91A}hSKcmRHSpo;JnFi#~UT$riJiR^YP6L|n6dqlCb!WR6(T-kc zG1}u?OsP$la#ppusD8{)E)E-*YBL+f@LbO9Oey;u>ok|W6thGcS*C4FoC;);BjxH7a={TyEKPOK zX~n(aZ29|kuf*OflnI!sG~Gi-)LNuTFVG;5U+8vJ-0k?RtF_Bf@RfQUt$o9#Dtu-^ z>g?*L6^5&3>B+w6+3+!IZn&RgAleM_DZTJJGuv?O$>Vu+d!i)wFcU3)#7p`n9^Da6 zUlH~h-RiG&R@B!_Q8c;A0vGmhXI_j5^F8)sLhcLsnE3B164r79#q_U|4q_XFC1~t5 z6nj;b%lF|H@+pxs5t@fErXM@Dy+0fg*)&}@Tt{@UT(TeZy)jjl_t~o~!MP5eVA#jU zSG#?c<>7MRYX^A;pAV|hV_#kUcBnojrt+Co1kQ}rmTNVN;S^y(;rhhNB#!Q+1tEZQsF)O@5OBcR+JFq>HS}98_DXCx zXh4Q*7m(y0V)$P`W?;+yx7Y>h1JIQj1eAF8)C~f5h%txxO=Jvu3kWDfh%?|wICD?Gb{?s|abzadych;K^JL9%5N!#;4O8;|fo zp=u`IbWc8?SlIo+0;;CzlHbl>AnkI{7^>z*uP0Rr=-lRV6z;e)&idU4MoNA6$cPg4 zoL(%|-b{5mz8=*5xbvQ;_OXr(4fj z)x`JJrqo}^$xM4(xn|X6+S?$U`*d#YN5r<0axdDCvC7?3 zv!4*pl{5|-B_xY^t*k9s7ue=hX6{4ipCd4H89n&OF_k|g5?(&E=eeTBLF0tdJg+Y4 zB+~2{^NgwIZsNzWg&E$V;{A!FS(eMjCu(q5EJm=WPeKp_cCVw<`^kV@_kB-*Brh5( zOer&OA6eX+P(85_^hynNUKnE-ex@oe^xKL7d45rI^l0;8B8U{FJLRko@R_NUd)O11 z!Jst0^RSh5$k12IT=l#$_ro?8SA90H=uRk+I&DeQ_2?UV>a_jAZ+GGG`~@o6d=l_< zN2rfVQ6@2+?rq-?-=C(*Tff+F&?hCF!iI9QLlsk4S2M< zFr}yo{~(MGpP86>ziGn%xD|BB#NObsI1^RgfoikIRt-kv{Gnu)q!nd6K9j#9IiN?; z4`}xD9xig+SqLa?5ngkiEJiOTNS!i(HJ_Mp5ulPMByEYgoMnn6kOye8=3Og2BeL(J z<6VqLZokA`d}amfteoaMssXDLjLr@2fI7}CZ<~%qF$+@^3sPHq&8g}(e@aTWIdllf zn~ijMb`ylqg^ti`dtWbuu_aB(h#(Yo3!Js0d18sAo5kpr+0lFu8;$YigKkX`B-Gr) z${#i&?cTJ)m{R4#`@j~5BplDUD14z#J3wL*#xQD}t;+fN70B}@NA_7M={wGa=utGF zL+0nMUarZ!{LF@;7SO)4$e5bV1{adqsb;j**j45&ObNea056pa1_5iJv8(udIG(@3 zG1BNE=Ge1vTy^wa?MnPzs(!C7mBPztz`QEs@YiK~g(=+EF7$Q6h%0$Q6mT)L6SvBD zP03p;P2vvQQ{Q9 zFbLpWEcd(w;6W|-Q@wctAC}`(U#-+J^6R(Uk}yn10j%X#HD>wETRnaW!&}F0c$E%3 zHzI`1yxx~%_3-r1kBFLP^0z|%KvQ48R!y){8KrF2USQWBO?tK%d zsj?uzH%dbR?)I};B(o_|%mNSaBFOo7qB0Fu)@Oz7yh@V#0Sf_KiJb{+GmZRhmDhL` zZhfwgF)XkxasAqvRZK+SzYR zah!VkCR`PnB7J*r%CL)1l%8Q{Lc0G4C@|o$+3{YN_$2=yADW>e>8>1&00BH@L}phU z9R$GBY}G1d*$_ZI#sq;YR7`N}0tefb&y}<7H-Q;V7+86Pj-Gy$kn9lL91^DL-y{RF zXSuh)YhjxK*?;8btBFb%Y}$8 zf)dxi$8|$ht#Ib$#8Rbe=BsxnVqkmC2&Fp{lqDM3!-6EW>mTnYEZhv}tQExI=(tX> zzSMkPPQ0Ir51yNNom>SVP3hoDYa*hkjzQNut1UytZoVI7x?kS&Hgwk%z?EhoDK!?k zCIme6CfFg7kfZxDOORJum0$|TX9n$Fd2_UoyYKRxD^4{x8i0u0<*^NU(Itl@eVw6T zXR9>AD?*RF7Wme!H#)(Q6!7K{1zcWxEh#DZHGmVosSKKps)1X61|Y4bG;s}t00e-f zvns2TUAggbyb30}PJWmAOt=+7v!fh9!}rBS>RjKP^dJw;YCx$1nB? z8g^wqSwXy)mt#TTO74!MuguMCI)TQm*r`L+*0_6%BekJ5;MPYduq&0DzyAFtOf(5A zPl$O}zAYWVS8u<9+u<>wt;7)wjwZ^s90@!8^j)FjP(S0eZYO~6MRU`Tu|(4^6w1O- zOKXY@{SDRbMR;zy+awcYXfWdY#e`LWWQzgm^7~$-LCQM6jMjbeJ_U{zVul~p;&Ig! zZsC%b+EBHRA!NM+eu;O#xgW-$Lr1xrRzUhG$P+uWE<$NCD(h8h0RZe#UHa{D#T!t@ zB<#G)bAqxjWzwp5>4k~xMo$f%g$#9#A1ZatQ45%~w@A-!o^4=Hej{{rN`*qu&x|6J zT(CwlpTh`=PVPTLpWOZE+7sM)>fdNStOg~Lel&{aaUIbmmwR76UjeSArz-q^3}*#~ zFjlPre0eapKMNgl@<}2cU{DzCNhE3Uw0@ohOcx*Gr=-dROYh_P3FmeNSQ7%c?S(Nhb*$xb z!juFCR9OWM2oWcq-09s^AV!Rly, message_op=) +``` + +As we can see, the `file_event` endpoint is associated with the Redis channel `public/file_event/samx` and the message type `FileMessage`. The message operations supported by this endpoint are `register`, `set_and_publish`, `delete`, `get`, and `keys`. + +In order to send a message to this endpoint, we need to create a new instance of the `FileMessage` class and send it to the Redis server using the `set_and_publish` operation: + +```python +from bec_lib.endpoints import MessageEndpoints +from bec_lib import messages + +# create a new instance of a BECMessage class, e.g. a FileMessage +msg = messages.FileMessage(file_path='path/to/file', done=False, successful=True, metadata={'scan_id': "1234"}) + +bec.connector.set_and_publish(MessageEndpoints.file_event("samx"), msg) + +``` + +Another service listening to the `public/file_event/samx` channel will receive the message and can act accordingly. +If we simply want to retrieve the last message from the `public/file_event/samx` channel, we can use the `get` operation: + +```python +from bec_lib.endpoints import MessageEndpoints + +# get the last message from the 'public/file_event/samx' channel +out = bec.connector.get(MessageEndpoints.file_event("samx")) +out +``` + +```` + +## BECMessage classes +The BECMessage classes are used to create messages that can be sent to Redis. Upon initialization, a BECMessage class validates the input against the predefined schema and raises an error in case of a mismatch. This ensures that the data sent to Redis is always in the correct format. The BECMessage classes are defined in the `bec_lib.messages`: + +````{dropdown} View code: BECMessage classes +```{literalinclude} ../../../../bec_lib/bec_lib/messages.py +``` +```` + +Every instance of a BECMessage provides the attributes `content` and `metadata`, both returning dictionaries. The `content` attribute contains the actual data that is sent to Redis, while the `metadata` attribute contains additional information about the message.