[lldb] [llgs] Move client-server communication into a separate thread (WIP)

[mgorny]

Summary

Perform all the communication between client and server in a dedicated
thread that's started after establishing the connection. The thread
uses a MainLoop instance to read all incoming packets into a queue,
and handle asynchronous requests from the client via
the AddPendingCallback() API.

Using a dedicated thread will make it possible to continuously read
asynchronous notifications from running processes while permitting
the plugin to simultaneously handle synchronous requests in multiprocess
mode.

Sponsored by: The FreeBSD Foundation

TODO:

• implement complete error handling
• implement timeouts

[labath]

I think we should get some measurements (e.g. from the process plugin packet speed-test cmd) of the overhead of this approach. The latency/RTT of the connection is very important for some users, and I fear that all of this ping pong could significantly regress that.

[mgorny]

In D135031#3833661, @labath wrote:

I think we should get some measurements (e.g. from the process plugin packet speed-test cmd) of the overhead of this approach. The latency/RTT of the connection is very important for some users, and I fear that all of this ping pong could significantly regress that.

Hmm, it seems that I'm seeing half the speed of the original. Not sure how much of a deal that is.

Before:

Testing sending 1000 packets of various sizes:
qSpeedTest(send=      0, recv=      0) in 0,032515690 s for  30754,38 packets/s (0,032515 ms per packet) with standard deviation of 0,002605 ms
qSpeedTest(send=      0, recv=      4) in 0,034152463 s for  29280,46 packets/s (0,034152 ms per packet) with standard deviation of 0,002709 ms
qSpeedTest(send=      0, recv=      8) in 0,036265273 s for  27574,59 packets/s (0,036265 ms per packet) with standard deviation of 0,001512 ms
qSpeedTest(send=      0, recv=     16) in 0,023616176 s for  42343,86 packets/s (0,023616 ms per packet) with standard deviation of 0,008221 ms
qSpeedTest(send=      0, recv=     32) in 0,019671151 s for  50835,86 packets/s (0,019671 ms per packet) with standard deviation of 0,000691 ms
qSpeedTest(send=      0, recv=     64) in 0,019893631 s for  50267,34 packets/s (0,019893 ms per packet) with standard deviation of 0,000811 ms
qSpeedTest(send=      0, recv=    128) in 0,020450350 s for  48898,92 packets/s (0,020450 ms per packet) with standard deviation of 0,001330 ms
qSpeedTest(send=      0, recv=    256) in 0,021703819 s for  46074,84 packets/s (0,021703 ms per packet) with standard deviation of 0,001262 ms
qSpeedTest(send=      0, recv=    512) in 0,023610808 s for  42353,48 packets/s (0,023610 ms per packet) with standard deviation of 0,001446 ms
qSpeedTest(send=      0, recv=   1024) in 0,027390972 s for  36508,38 packets/s (0,027390 ms per packet) with standard deviation of 0,002319 ms
qSpeedTest(send=      4, recv=      0) in 0,018693436 s for  53494,71 packets/s (0,018693 ms per packet) with standard deviation of 0,001463 ms
qSpeedTest(send=      4, recv=      4) in 0,018819677 s for  53135,88 packets/s (0,018819 ms per packet) with standard deviation of 0,001217 ms
qSpeedTest(send=      4, recv=      8) in 0,020369085 s for  49094,01 packets/s (0,020369 ms per packet) with standard deviation of 0,001513 ms
qSpeedTest(send=      4, recv=     16) in 0,020784404 s for  48113,00 packets/s (0,020784 ms per packet) with standard deviation of 0,002064 ms
qSpeedTest(send=      4, recv=     32) in 0,020945720 s for  47742,45 packets/s (0,020945 ms per packet) with standard deviation of 0,002698 ms
qSpeedTest(send=      4, recv=     64) in 0,023253340 s for  43004,57 packets/s (0,023253 ms per packet) with standard deviation of 0,002361 ms
qSpeedTest(send=      4, recv=    128) in 0,022991221 s for  43494,86 packets/s (0,022991 ms per packet) with standard deviation of 0,002397 ms
qSpeedTest(send=      4, recv=    256) in 0,024546532 s for  40738,95 packets/s (0,024546 ms per packet) with standard deviation of 0,001291 ms
qSpeedTest(send=      4, recv=    512) in 0,026892496 s for  37185,09 packets/s (0,026892 ms per packet) with standard deviation of 0,001912 ms
qSpeedTest(send=      4, recv=   1024) in 0,031360906 s for  31886,83 packets/s (0,031360 ms per packet) with standard deviation of 0,003003 ms
qSpeedTest(send=      8, recv=      0) in 0,020722076 s for  48257,71 packets/s (0,020722 ms per packet) with standard deviation of 0,002566 ms
qSpeedTest(send=      8, recv=      4) in 0,022102837 s for  45243,06 packets/s (0,022102 ms per packet) with standard deviation of 0,001933 ms
qSpeedTest(send=      8, recv=      8) in 0,021921420 s for  45617,48 packets/s (0,021921 ms per packet) with standard deviation of 0,001025 ms
qSpeedTest(send=      8, recv=     16) in 0,019039322 s for  52522,88 packets/s (0,019039 ms per packet) with standard deviation of 0,002766 ms
qSpeedTest(send=      8, recv=     32) in 0,019113524 s for  52318,98 packets/s (0,019113 ms per packet) with standard deviation of 0,003287 ms
qSpeedTest(send=      8, recv=     64) in 0,018969281 s for  52716,81 packets/s (0,018969 ms per packet) with standard deviation of 0,000611 ms
qSpeedTest(send=      8, recv=    128) in 0,019544501 s for  51165,29 packets/s (0,019544 ms per packet) with standard deviation of 0,000706 ms
qSpeedTest(send=      8, recv=    256) in 0,020592488 s for  48561,40 packets/s (0,020592 ms per packet) with standard deviation of 0,000887 ms
qSpeedTest(send=      8, recv=    512) in 0,022484412 s for  44475,26 packets/s (0,022484 ms per packet) with standard deviation of 0,000715 ms
qSpeedTest(send=      8, recv=   1024) in 0,023713522 s for  42170,04 packets/s (0,023713 ms per packet) with standard deviation of 0,003766 ms
qSpeedTest(send=     16, recv=      0) in 0,014724865 s for  67912,34 packets/s (0,014724 ms per packet) with standard deviation of 0,001221 ms
qSpeedTest(send=     16, recv=      4) in 0,014882473 s for  67193,13 packets/s (0,014882 ms per packet) with standard deviation of 0,000586 ms
qSpeedTest(send=     16, recv=      8) in 0,015224864 s for  65682,03 packets/s (0,015224 ms per packet) with standard deviation of 0,001100 ms
qSpeedTest(send=     16, recv=     16) in 0,015331225 s for  65226,36 packets/s (0,015331 ms per packet) with standard deviation of 0,000719 ms
qSpeedTest(send=     16, recv=     32) in 0,015794843 s for  63311,80 packets/s (0,015794 ms per packet) with standard deviation of 0,001211 ms
qSpeedTest(send=     16, recv=     64) in 0,015735803 s for  63549,35 packets/s (0,015735 ms per packet) with standard deviation of 0,000491 ms
qSpeedTest(send=     16, recv=    128) in 0,016455669 s for  60769,33 packets/s (0,016455 ms per packet) with standard deviation of 0,001518 ms
qSpeedTest(send=     16, recv=    256) in 0,016706374 s for  59857,39 packets/s (0,016706 ms per packet) with standard deviation of 0,000804 ms
qSpeedTest(send=     16, recv=    512) in 0,018495832 s for  54066,23 packets/s (0,018495 ms per packet) with standard deviation of 0,001457 ms
qSpeedTest(send=     16, recv=   1024) in 0,020938426 s for  47759,08 packets/s (0,020938 ms per packet) with standard deviation of 0,000632 ms
qSpeedTest(send=     32, recv=      0) in 0,014937217 s for  66946,88 packets/s (0,014937 ms per packet) with standard deviation of 0,001230 ms
qSpeedTest(send=     32, recv=      4) in 0,015045843 s for  66463,54 packets/s (0,015045 ms per packet) with standard deviation of 0,000459 ms
qSpeedTest(send=     32, recv=      8) in 0,015475038 s for  64620,20 packets/s (0,015475 ms per packet) with standard deviation of 0,001281 ms
qSpeedTest(send=     32, recv=     16) in 0,015376571 s for  65034,01 packets/s (0,015376 ms per packet) with standard deviation of 0,000441 ms
qSpeedTest(send=     32, recv=     32) in 0,016108289 s for  62079,84 packets/s (0,016108 ms per packet) with standard deviation of 0,001244 ms
qSpeedTest(send=     32, recv=     64) in 0,016165899 s for  61858,61 packets/s (0,016165 ms per packet) with standard deviation of 0,000565 ms
qSpeedTest(send=     32, recv=    128) in 0,016683761 s for  59938,52 packets/s (0,016683 ms per packet) with standard deviation of 0,001309 ms
qSpeedTest(send=     32, recv=    256) in 0,017112507 s for  58436,79 packets/s (0,017112 ms per packet) with standard deviation of 0,001249 ms
qSpeedTest(send=     32, recv=    512) in 0,018683286 s for  53523,77 packets/s (0,018683 ms per packet) with standard deviation of 0,001037 ms
qSpeedTest(send=     32, recv=   1024) in 0,021408148 s for  46711,19 packets/s (0,021408 ms per packet) with standard deviation of 0,001632 ms
qSpeedTest(send=     64, recv=      0) in 0,014771803 s for  67696,54 packets/s (0,014771 ms per packet) with standard deviation of 0,000465 ms
qSpeedTest(send=     64, recv=      4) in 0,015452625 s for  64713,92 packets/s (0,015452 ms per packet) with standard deviation of 0,001260 ms
qSpeedTest(send=     64, recv=      8) in 0,015246284 s for  65589,75 packets/s (0,015246 ms per packet) with standard deviation of 0,000576 ms
qSpeedTest(send=     64, recv=     16) in 0,016167672 s for  61851,82 packets/s (0,016167 ms per packet) with standard deviation of 0,002055 ms
qSpeedTest(send=     64, recv=     32) in 0,015814994 s for  63231,14 packets/s (0,015814 ms per packet) with standard deviation of 0,000593 ms
qSpeedTest(send=     64, recv=     64) in 0,016694322 s for  59900,61 packets/s (0,016694 ms per packet) with standard deviation of 0,002493 ms
qSpeedTest(send=     64, recv=    128) in 0,016346602 s for  61174,79 packets/s (0,016346 ms per packet) with standard deviation of 0,000689 ms
qSpeedTest(send=     64, recv=    256) in 0,017390292 s for  57503,35 packets/s (0,017390 ms per packet) with standard deviation of 0,002014 ms
qSpeedTest(send=     64, recv=    512) in 0,018724425 s for  53406,18 packets/s (0,018724 ms per packet) with standard deviation of 0,001452 ms
qSpeedTest(send=     64, recv=   1024) in 0,021263914 s for  47028,03 packets/s (0,021263 ms per packet) with standard deviation of 0,001241 ms
qSpeedTest(send=    128, recv=      0) in 0,015573244 s for  64212,70 packets/s (0,015573 ms per packet) with standard deviation of 0,001546 ms
qSpeedTest(send=    128, recv=      4) in 0,015465640 s for  64659,46 packets/s (0,015465 ms per packet) with standard deviation of 0,000543 ms
qSpeedTest(send=    128, recv=      8) in 0,015961170 s for  62652,05 packets/s (0,015961 ms per packet) with standard deviation of 0,001500 ms
qSpeedTest(send=    128, recv=     16) in 0,015820574 s for  63208,83 packets/s (0,015820 ms per packet) with standard deviation of 0,000546 ms
qSpeedTest(send=    128, recv=     32) in 0,016328989 s for  61240,78 packets/s (0,016328 ms per packet) with standard deviation of 0,001549 ms
qSpeedTest(send=    128, recv=     64) in 0,016259447 s for  61502,71 packets/s (0,016259 ms per packet) with standard deviation of 0,000537 ms
qSpeedTest(send=    128, recv=    128) in 0,016924068 s for  59087,45 packets/s (0,016924 ms per packet) with standard deviation of 0,001568 ms
qSpeedTest(send=    128, recv=    256) in 0,018336987 s for  54534,59 packets/s (0,018336 ms per packet) with standard deviation of 0,002028 ms
qSpeedTest(send=    128, recv=    512) in 0,019908693 s for  50229,32 packets/s (0,019908 ms per packet) with standard deviation of 0,003402 ms
qSpeedTest(send=    128, recv=   1024) in 0,023228262 s for  43051,00 packets/s (0,023228 ms per packet) with standard deviation of 0,002937 ms
qSpeedTest(send=    256, recv=      0) in 0,015928367 s for  62781,07 packets/s (0,015928 ms per packet) with standard deviation of 0,000877 ms
qSpeedTest(send=    256, recv=      4) in 0,018923203 s for  52845,18 packets/s (0,018923 ms per packet) with standard deviation of 0,002864 ms
qSpeedTest(send=    256, recv=      8) in 0,019888291 s for  50280,84 packets/s (0,019888 ms per packet) with standard deviation of 0,001989 ms
qSpeedTest(send=    256, recv=     16) in 0,019919142 s for  50202,96 packets/s (0,019919 ms per packet) with standard deviation of 0,001227 ms
qSpeedTest(send=    256, recv=     32) in 0,020463344 s for  48867,87 packets/s (0,020463 ms per packet) with standard deviation of 0,002039 ms
qSpeedTest(send=    256, recv=     64) in 0,021069687 s for  47461,55 packets/s (0,021069 ms per packet) with standard deviation of 0,002684 ms
qSpeedTest(send=    256, recv=    128) in 0,021204460 s for  47159,89 packets/s (0,021204 ms per packet) with standard deviation of 0,000749 ms
qSpeedTest(send=    256, recv=    256) in 0,023324074 s for  42874,16 packets/s (0,023324 ms per packet) with standard deviation of 0,002311 ms
qSpeedTest(send=    256, recv=    512) in 0,024667764 s for  40538,74 packets/s (0,024667 ms per packet) with standard deviation of 0,002092 ms
qSpeedTest(send=    256, recv=   1024) in 0,029726071 s for  33640,50 packets/s (0,029726 ms per packet) with standard deviation of 0,002336 ms
qSpeedTest(send=    512, recv=      0) in 0,019497503 s for  51288,62 packets/s (0,019497 ms per packet) with standard deviation of 0,014002 ms
qSpeedTest(send=    512, recv=      4) in 0,017419128 s for  57408,16 packets/s (0,017419 ms per packet) with standard deviation of 0,001145 ms
qSpeedTest(send=    512, recv=      8) in 0,017606512 s for  56797,17 packets/s (0,017606 ms per packet) with standard deviation of 0,000878 ms
qSpeedTest(send=    512, recv=     16) in 0,018146981 s for  55105,59 packets/s (0,018146 ms per packet) with standard deviation of 0,001115 ms
qSpeedTest(send=    512, recv=     32) in 0,018310759 s for  54612,70 packets/s (0,018310 ms per packet) with standard deviation of 0,001008 ms
qSpeedTest(send=    512, recv=     64) in 0,018420037 s for  54288,71 packets/s (0,018420 ms per packet) with standard deviation of 0,000794 ms
qSpeedTest(send=    512, recv=    128) in 0,018898545 s for  52914,12 packets/s (0,018898 ms per packet) with standard deviation of 0,000968 ms
qSpeedTest(send=    512, recv=    256) in 0,019524993 s for  51216,41 packets/s (0,019524 ms per packet) with standard deviation of 0,001062 ms
qSpeedTest(send=    512, recv=    512) in 0,021158433 s for  47262,48 packets/s (0,021158 ms per packet) with standard deviation of 0,000788 ms
qSpeedTest(send=    512, recv=   1024) in 0,023771813 s for  42066,63 packets/s (0,023771 ms per packet) with standard deviation of 0,000909 ms
qSpeedTest(send=   1024, recv=      0) in 0,020384476 s for  49056,94 packets/s (0,020384 ms per packet) with standard deviation of 0,001174 ms
qSpeedTest(send=   1024, recv=      4) in 0,020033369 s for  49916,72 packets/s (0,020033 ms per packet) with standard deviation of 0,001284 ms
qSpeedTest(send=   1024, recv=      8) in 0,019968877 s for  50077,93 packets/s (0,019968 ms per packet) with standard deviation of 0,000994 ms
qSpeedTest(send=   1024, recv=     16) in 0,020178104 s for  49558,67 packets/s (0,020178 ms per packet) with standard deviation of 0,000815 ms
qSpeedTest(send=   1024, recv=     32) in 0,022866154 s for  43732,76 packets/s (0,022866 ms per packet) with standard deviation of 0,004857 ms
qSpeedTest(send=   1024, recv=     64) in 0,031659000 s for  31586,60 packets/s (0,031659 ms per packet) with standard deviation of 0,001603 ms
qSpeedTest(send=   1024, recv=    128) in 0,032057628 s for  31193,82 packets/s (0,032057 ms per packet) with standard deviation of 0,001786 ms
qSpeedTest(send=   1024, recv=    256) in 0,024415923 s for  40956,88 packets/s (0,024415 ms per packet) with standard deviation of 0,003273 ms
qSpeedTest(send=   1024, recv=    512) in 0,029499674 s for  33898,68 packets/s (0,029499 ms per packet) with standard deviation of 0,010539 ms
qSpeedTest(send=   1024, recv=   1024) in 0,032547455 s for  30724,37 packets/s (0,032547 ms per packet) with standard deviation of 0,003222 ms
Testing receiving 4,0MB of data using varying receive packet sizes:
qSpeedTest(send=      0, recv=     32) 131072 packets needed to receive 4,0MB in 2,463062286 s for 1,62 MB/sec for  53215,06 packets/sec (0,018791 ms per packet)
qSpeedTest(send=      0, recv=     64)  65536 packets needed to receive 4,0MB in 1,288395047 s for 3,10 MB/sec for  50866,39 packets/sec (0,019659 ms per packet)
qSpeedTest(send=      0, recv=    128)  32768 packets needed to receive 4,0MB in 0,677966475 s for 5,90 MB/sec for  48332,77 packets/sec (0,020689 ms per packet)
qSpeedTest(send=      0, recv=    256)  16384 packets needed to receive 4,0MB in 0,300569832 s for 13,31 MB/sec for  54509,80 packets/sec (0,018345 ms per packet)
qSpeedTest(send=      0, recv=    512)   8192 packets needed to receive 4,0MB in 0,161001429 s for 24,84 MB/sec for  50881,54 packets/sec (0,019653 ms per packet)
qSpeedTest(send=      0, recv=   1024)   4096 packets needed to receive 4,0MB in 0,091886751 s for 43,53 MB/sec for  44576,61 packets/sec (0,022433 ms per packet)

After:

```Testing sending 1000 packets of various sizes:
qSpeedTest(send=      0, recv=      0) in 0,038984288 s for  25651,36 packets/s (0,038984 ms per packet) with standard deviation of 0,005802 ms
qSpeedTest(send=      0, recv=      4) in 0,052455731 s for  19063,69 packets/s (0,052455 ms per packet) with standard deviation of 0,002935 ms
qSpeedTest(send=      0, recv=      8) in 0,053561039 s for  18670,29 packets/s (0,053561 ms per packet) with standard deviation of 0,002780 ms
qSpeedTest(send=      0, recv=     16) in 0,054524228 s for  18340,47 packets/s (0,054524 ms per packet) with standard deviation of 0,002989 ms
qSpeedTest(send=      0, recv=     32) in 0,054595515 s for  18316,52 packets/s (0,054595 ms per packet) with standard deviation of 0,003006 ms
qSpeedTest(send=      0, recv=     64) in 0,054095767 s for  18485,73 packets/s (0,054095 ms per packet) with standard deviation of 0,004307 ms
qSpeedTest(send=      0, recv=    128) in 0,054310042 s for  18412,80 packets/s (0,054310 ms per packet) with standard deviation of 0,003742 ms
qSpeedTest(send=      0, recv=    256) in 0,055870242 s for  17898,62 packets/s (0,055870 ms per packet) with standard deviation of 0,004206 ms
qSpeedTest(send=      0, recv=    512) in 0,050346799 s for  19862,24 packets/s (0,050346 ms per packet) with standard deviation of 0,004942 ms
qSpeedTest(send=      0, recv=   1024) in 0,052045114 s for  19214,10 packets/s (0,052045 ms per packet) with standard deviation of 0,002085 ms
qSpeedTest(send=      4, recv=      0) in 0,043030608 s for  23239,27 packets/s (0,043030 ms per packet) with standard deviation of 0,001047 ms
qSpeedTest(send=      4, recv=      4) in 0,042633153 s for  23455,92 packets/s (0,042633 ms per packet) with standard deviation of 0,002444 ms
qSpeedTest(send=      4, recv=      8) in 0,035519697 s for  28153,39 packets/s (0,035519 ms per packet) with standard deviation of 0,001563 ms
qSpeedTest(send=      4, recv=     16) in 0,035472117 s for  28191,16 packets/s (0,035472 ms per packet) with standard deviation of 0,000954 ms
qSpeedTest(send=      4, recv=     32) in 0,035710789 s for  28002,74 packets/s (0,035710 ms per packet) with standard deviation of 0,000966 ms
qSpeedTest(send=      4, recv=     64) in 0,036006186 s for  27773,01 packets/s (0,036006 ms per packet) with standard deviation of 0,000926 ms
qSpeedTest(send=      4, recv=    128) in 0,036690224 s for  27255,22 packets/s (0,036690 ms per packet) with standard deviation of 0,001000 ms
qSpeedTest(send=      4, recv=    256) in 0,037529159 s for  26645,95 packets/s (0,037529 ms per packet) with standard deviation of 0,001083 ms
qSpeedTest(send=      4, recv=    512) in 0,039512560 s for  25308,41 packets/s (0,039512 ms per packet) with standard deviation of 0,000955 ms
qSpeedTest(send=      4, recv=   1024) in 0,043194912 s for  23150,88 packets/s (0,043194 ms per packet) with standard deviation of 0,001877 ms
qSpeedTest(send=      8, recv=      0) in 0,034951087 s for  28611,41 packets/s (0,034951 ms per packet) with standard deviation of 0,001792 ms
qSpeedTest(send=      8, recv=      4) in 0,036084294 s for  27712,89 packets/s (0,036084 ms per packet) with standard deviation of 0,002015 ms
qSpeedTest(send=      8, recv=      8) in 0,035335865 s for  28299,86 packets/s (0,035335 ms per packet) with standard deviation of 0,001496 ms
qSpeedTest(send=      8, recv=     16) in 0,038773648 s for  25790,71 packets/s (0,038773 ms per packet) with standard deviation of 0,002936 ms
qSpeedTest(send=      8, recv=     32) in 0,041695233 s for  23983,56 packets/s (0,041695 ms per packet) with standard deviation of 0,003830 ms
qSpeedTest(send=      8, recv=     64) in 0,038607225 s for  25901,89 packets/s (0,038607 ms per packet) with standard deviation of 0,009730 ms
qSpeedTest(send=      8, recv=    128) in 0,047023695 s for  21265,88 packets/s (0,047023 ms per packet) with standard deviation of 0,003495 ms
qSpeedTest(send=      8, recv=    256) in 0,050027635 s for  19988,95 packets/s (0,050027 ms per packet) with standard deviation of 0,001358 ms
qSpeedTest(send=      8, recv=    512) in 0,054041311 s for  18504,36 packets/s (0,054041 ms per packet) with standard deviation of 0,005053 ms
qSpeedTest(send=      8, recv=   1024) in 0,056729123 s for  17627,63 packets/s (0,056729 ms per packet) with standard deviation of 0,013785 ms
qSpeedTest(send=     16, recv=      0) in 0,030867426 s for  32396,61 packets/s (0,030867 ms per packet) with standard deviation of 0,001643 ms
qSpeedTest(send=     16, recv=      4) in 0,031702854 s for  31542,90 packets/s (0,031702 ms per packet) with standard deviation of 0,001906 ms
qSpeedTest(send=     16, recv=      8) in 0,031275541 s for  31973,87 packets/s (0,031275 ms per packet) with standard deviation of 0,002191 ms
qSpeedTest(send=     16, recv=     16) in 0,031657897 s for  31587,70 packets/s (0,031657 ms per packet) with standard deviation of 0,001185 ms
qSpeedTest(send=     16, recv=     32) in 0,031606868 s for  31638,69 packets/s (0,031606 ms per packet) with standard deviation of 0,001112 ms
qSpeedTest(send=     16, recv=     64) in 0,031989716 s for  31260,05 packets/s (0,031989 ms per packet) with standard deviation of 0,001429 ms
qSpeedTest(send=     16, recv=    128) in 0,032680295 s for  30599,48 packets/s (0,032680 ms per packet) with standard deviation of 0,000935 ms
qSpeedTest(send=     16, recv=    256) in 0,035144683 s for  28453,81 packets/s (0,035144 ms per packet) with standard deviation of 0,004727 ms
qSpeedTest(send=     16, recv=    512) in 0,052230734 s for  19145,81 packets/s (0,052230 ms per packet) with standard deviation of 0,002293 ms
qSpeedTest(send=     16, recv=   1024) in 0,055264391 s for  18094,83 packets/s (0,055264 ms per packet) with standard deviation of 0,002330 ms
qSpeedTest(send=     32, recv=      0) in 0,046223037 s for  21634,23 packets/s (0,046223 ms per packet) with standard deviation of 0,002007 ms
qSpeedTest(send=     32, recv=      4) in 0,047726098 s for  20952,90 packets/s (0,047726 ms per packet) with standard deviation of 0,002343 ms
qSpeedTest(send=     32, recv=      8) in 0,047792483 s for  20923,79 packets/s (0,047792 ms per packet) with standard deviation of 0,002122 ms
qSpeedTest(send=     32, recv=     16) in 0,048320290 s for  20695,24 packets/s (0,048320 ms per packet) with standard deviation of 0,001856 ms
qSpeedTest(send=     32, recv=     32) in 0,048498556 s for  20619,17 packets/s (0,048498 ms per packet) with standard deviation of 0,002041 ms
qSpeedTest(send=     32, recv=     64) in 0,048730254 s for  20521,13 packets/s (0,048730 ms per packet) with standard deviation of 0,002181 ms
qSpeedTest(send=     32, recv=    128) in 0,049700897 s for  20120,36 packets/s (0,049700 ms per packet) with standard deviation of 0,002352 ms
qSpeedTest(send=     32, recv=    256) in 0,050448231 s for  19822,30 packets/s (0,050448 ms per packet) with standard deviation of 0,003351 ms
qSpeedTest(send=     32, recv=    512) in 0,049088776 s for  20371,26 packets/s (0,049088 ms per packet) with standard deviation of 0,007308 ms
qSpeedTest(send=     32, recv=   1024) in 0,056962457 s for  17555,42 packets/s (0,056962 ms per packet) with standard deviation of 0,010280 ms
qSpeedTest(send=     64, recv=      0) in 0,052351017 s for  19101,83 packets/s (0,052351 ms per packet) with standard deviation of 0,003792 ms
qSpeedTest(send=     64, recv=      4) in 0,046745479 s for  21392,44 packets/s (0,046745 ms per packet) with standard deviation of 0,006138 ms
qSpeedTest(send=     64, recv=      8) in 0,057475079 s for  17398,85 packets/s (0,057475 ms per packet) with standard deviation of 0,004685 ms
qSpeedTest(send=     64, recv=     16) in 0,058971055 s for  16957,47 packets/s (0,058971 ms per packet) with standard deviation of 0,004321 ms
qSpeedTest(send=     64, recv=     32) in 0,061164167 s for  16349,44 packets/s (0,061164 ms per packet) with standard deviation of 0,002456 ms
qSpeedTest(send=     64, recv=     64) in 0,059380196 s for  16840,63 packets/s (0,059380 ms per packet) with standard deviation of 0,004481 ms
qSpeedTest(send=     64, recv=    128) in 0,050775044 s for  19694,71 packets/s (0,050775 ms per packet) with standard deviation of 0,013567 ms
qSpeedTest(send=     64, recv=    256) in 0,047020558 s for  21267,29 packets/s (0,047020 ms per packet) with standard deviation of 0,005597 ms
qSpeedTest(send=     64, recv=    512) in 0,051683728 s for  19348,45 packets/s (0,051683 ms per packet) with standard deviation of 0,002022 ms
qSpeedTest(send=     64, recv=   1024) in 0,054888368 s for  18218,80 packets/s (0,054888 ms per packet) with standard deviation of 0,002123 ms
qSpeedTest(send=    128, recv=      0) in 0,046791244 s for  21371,52 packets/s (0,046791 ms per packet) with standard deviation of 0,001900 ms
qSpeedTest(send=    128, recv=      4) in 0,047672436 s for  20976,48 packets/s (0,047672 ms per packet) with standard deviation of 0,001848 ms
qSpeedTest(send=    128, recv=      8) in 0,047766674 s for  20935,10 packets/s (0,047766 ms per packet) with standard deviation of 0,001761 ms
qSpeedTest(send=    128, recv=     16) in 0,048285920 s for  20709,97 packets/s (0,048285 ms per packet) with standard deviation of 0,001714 ms
qSpeedTest(send=    128, recv=     32) in 0,048339885 s for  20686,85 packets/s (0,048339 ms per packet) with standard deviation of 0,001668 ms
qSpeedTest(send=    128, recv=     64) in 0,048721701 s for  20524,73 packets/s (0,048721 ms per packet) with standard deviation of 0,001698 ms
qSpeedTest(send=    128, recv=    128) in 0,044173706 s for  22637,90 packets/s (0,044173 ms per packet) with standard deviation of 0,005995 ms
qSpeedTest(send=    128, recv=    256) in 0,044957425 s for  22243,27 packets/s (0,044957 ms per packet) with standard deviation of 0,005970 ms
qSpeedTest(send=    128, recv=    512) in 0,049801286 s for  20079,80 packets/s (0,049801 ms per packet) with standard deviation of 0,004223 ms
qSpeedTest(send=    128, recv=   1024) in 0,055070031 s for  18158,70 packets/s (0,055070 ms per packet) with standard deviation of 0,002434 ms
qSpeedTest(send=    256, recv=      0) in 0,047683310 s for  20971,70 packets/s (0,047683 ms per packet) with standard deviation of 0,002041 ms
qSpeedTest(send=    256, recv=      4) in 0,048382293 s for  20668,72 packets/s (0,048382 ms per packet) with standard deviation of 0,002112 ms
qSpeedTest(send=    256, recv=      8) in 0,048400559 s for  20660,92 packets/s (0,048400 ms per packet) with standard deviation of 0,002436 ms
qSpeedTest(send=    256, recv=     16) in 0,048583377 s for  20583,17 packets/s (0,048583 ms per packet) with standard deviation of 0,001701 ms
qSpeedTest(send=    256, recv=     32) in 0,048700549 s for  20533,65 packets/s (0,048700 ms per packet) with standard deviation of 0,001786 ms
qSpeedTest(send=    256, recv=     64) in 0,048930537 s for  20437,13 packets/s (0,048930 ms per packet) with standard deviation of 0,001881 ms
qSpeedTest(send=    256, recv=    128) in 0,049667772 s for  20133,78 packets/s (0,049667 ms per packet) with standard deviation of 0,001872 ms
qSpeedTest(send=    256, recv=    256) in 0,048879776 s for  20458,36 packets/s (0,048879 ms per packet) with standard deviation of 0,002454 ms
qSpeedTest(send=    256, recv=    512) in 0,051021773 s for  19599,47 packets/s (0,051021 ms per packet) with standard deviation of 0,002511 ms
qSpeedTest(send=    256, recv=   1024) in 0,055424366 s for  18042,61 packets/s (0,055424 ms per packet) with standard deviation of 0,002134 ms
qSpeedTest(send=    512, recv=      0) in 0,049034167 s for  20393,94 packets/s (0,049034 ms per packet) with standard deviation of 0,001815 ms
qSpeedTest(send=    512, recv=      4) in 0,050176170 s for  19929,78 packets/s (0,050176 ms per packet) with standard deviation of 0,002066 ms
qSpeedTest(send=    512, recv=      8) in 0,050845414 s for  19667,46 packets/s (0,050845 ms per packet) with standard deviation of 0,001791 ms
qSpeedTest(send=    512, recv=     16) in 0,050700471 s for  19723,68 packets/s (0,050700 ms per packet) with standard deviation of 0,001905 ms
qSpeedTest(send=    512, recv=     32) in 0,048288364 s for  20708,92 packets/s (0,048288 ms per packet) with standard deviation of 0,002969 ms
qSpeedTest(send=    512, recv=     64) in 0,048271544 s for  20716,14 packets/s (0,048271 ms per packet) with standard deviation of 0,001781 ms
qSpeedTest(send=    512, recv=    128) in 0,049687181 s for  20125,92 packets/s (0,049687 ms per packet) with standard deviation of 0,002522 ms
qSpeedTest(send=    512, recv=    256) in 0,051946048 s for  19250,74 packets/s (0,051946 ms per packet) with standard deviation of 0,003335 ms
qSpeedTest(send=    512, recv=    512) in 0,052447148 s for  19066,81 packets/s (0,052447 ms per packet) with standard deviation of 0,004008 ms
qSpeedTest(send=    512, recv=   1024) in 0,057615165 s for  17356,54 packets/s (0,057615 ms per packet) with standard deviation of 0,002161 ms
qSpeedTest(send=   1024, recv=      0) in 0,053596254 s for  18658,02 packets/s (0,053596 ms per packet) with standard deviation of 0,002452 ms
qSpeedTest(send=   1024, recv=      4) in 0,054260008 s for  18429,78 packets/s (0,054260 ms per packet) with standard deviation of 0,002081 ms
qSpeedTest(send=   1024, recv=      8) in 0,054516576 s for  18343,04 packets/s (0,054516 ms per packet) with standard deviation of 0,001920 ms
qSpeedTest(send=   1024, recv=     16) in 0,054831583 s for  18237,66 packets/s (0,054831 ms per packet) with standard deviation of 0,001952 ms
qSpeedTest(send=   1024, recv=     32) in 0,054993141 s for  18184,09 packets/s (0,054993 ms per packet) with standard deviation of 0,001813 ms
qSpeedTest(send=   1024, recv=     64) in 0,055353913 s for  18065,57 packets/s (0,055353 ms per packet) with standard deviation of 0,002061 ms
qSpeedTest(send=   1024, recv=    128) in 0,056008182 s for  17854,53 packets/s (0,056008 ms per packet) with standard deviation of 0,002051 ms
qSpeedTest(send=   1024, recv=    256) in 0,056844413 s for  17591,88 packets/s (0,056844 ms per packet) with standard deviation of 0,001971 ms
qSpeedTest(send=   1024, recv=    512) in 0,058375575 s for  17130,45 packets/s (0,058375 ms per packet) with standard deviation of 0,001795 ms
qSpeedTest(send=   1024, recv=   1024) in 0,061647356 s for  16221,30 packets/s (0,061647 ms per packet) with standard deviation of 0,002107 ms
Testing receiving 4,0MB of data using varying receive packet sizes:
qSpeedTest(send=      0, recv=     32) 131072 packets needed to receive 4,0MB in 6,461872101 s for 0,62 MB/sec for  20283,90 packets/sec (0,049300 ms per packet)
qSpeedTest(send=      0, recv=     64)  65536 packets needed to receive 4,0MB in 2,903709650 s for 1,38 MB/sec for  22569,75 packets/sec (0,044307 ms per packet)
qSpeedTest(send=      0, recv=    128)  32768 packets needed to receive 4,0MB in 1,473043919 s for 2,72 MB/sec for  22245,09 packets/sec (0,044953 ms per packet)
qSpeedTest(send=      0, recv=    256)  16384 packets needed to receive 4,0MB in 0,727462530 s for 5,50 MB/sec for  22522,12 packets/sec (0,044400 ms per packet)
qSpeedTest(send=      0, recv=    512)   8192 packets needed to receive 4,0MB in 0,373667091 s for 10,70 MB/sec for  21923,26 packets/sec (0,045613 ms per packet)
qSpeedTest(send=      0, recv=   1024)   4096 packets needed to receive 4,0MB in 0,182141349 s for 21,96 MB/sec for  22488,03 packets/sec (0,044468 ms per packet)

[mgorny]

For the record, I'm still hitting a lot of problems with this, including comm hanging sometimes (especially under load), some Python tests segfaulting with completely useless backtrace (pretty normal for Python backtraces), lots of pexpect-based test failures that also tend not to provide any useful output. @labath, if you happen to have an idea what could be wrong (besides what's marked TODO), I'd appreciate some suggestions. I am somewhat suspicious that the broadcaster/listener logic may not be bulletproof too.

[labath]

FWIW, I tried the to run the benchmark on an apple machine and got roughly the same results (although I never managed to get it to run to completion). I also don't know how much of a deal that is. I think someone from Apple ought to comment on that.

However, I have to say that this API feels very unergonomic. I still haven't managed to figure out how is it supposed to work, much less where the bug is. I think it's just operating at too low level. Instead of dealing with selects and callbacks, I'd say that the interface between the threads should be "this is the packet I want you to send, and here's is how I would like to get the reply".

lldb/source/Plugins/Process/gdb-remote/GDBRemoteClientBase.h

 //===-- GDBRemoteClientBase.h -----------------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLDB_SOURCE_PLUGINS_PROCESS_GDB_REMOTE_GDBREMOTECLIENTBASE_H
 #define LLDB_SOURCE_PLUGINS_PROCESS_GDB_REMOTE_GDBREMOTECLIENTBASE_H
 
 #include "GDBRemoteCommunication.h"
 
+#include "lldb/Host/HostThread.h"
+#include "lldb/Host/MainLoop.h"
+
 #include <condition_variable>
+#include <deque>
 
 namespace lldb_private {
 namespace process_gdb_remote {
 
 class GDBRemoteClientBase : public Broadcaster {
 public:
   typedef GDBRemoteCommunication::PacketResult PacketResult;
 
   enum {
     eBroadcastBitRunPacketSent = (1u << 0),
+    eBroadcastBitCommDone = (1u << 1),
+    eBroadcastBitCommThreadExited = (1u << 2),
+    eBroadcastBitCommPacketRecv = (1u << 3),
   };
 
   struct ContinueDelegate {
     virtual ~ContinueDelegate();
     virtual void HandleAsyncStdout(llvm::StringRef out) = 0;
     virtual void HandleAsyncMisc(llvm::StringRef data) = 0;
     virtual void HandleStopReply() = 0;
 
public:
   bool IsConnected() const;
 
   lldb::ConnectionStatus Disconnect(Status *error_ptr = nullptr);
 
   GDBRemoteCommunication &GetCommunication() {
     return m_comm;
   }
 
+  llvm::Error StartThread();
+  void StopThread();
+
 protected:
   GDBRemoteCommunication m_comm;
 
+  HostThread m_comm_thread;
+  bool m_comm_thread_exited;
+  std::unique_ptr<MainLoop> m_comm_loop_up;
+  std::deque<std::string> m_comm_sync_packet_queue;
+
+  void CommThreadReadHandler(MainLoopBase &loop);
+  lldb::thread_result_t CommThread();
+
+  bool RequestComm(const MainLoop::Callback &send_callback,
+                   const MainLoop::Callback &recv_callback);
+
   PacketResult
   SendPacketAndWaitForResponseNoLock(llvm::StringRef payload,
                                      StringExtractorGDBRemote &response);
 
   virtual void OnRunPacketSent(bool first);
 
 private:
   /// Variables handling synchronization between the Continue thread and any

lldb/source/Plugins/Process/gdb-remote/GDBRemoteClientBase.cpp

 //===-- GDBRemoteClientBase.cpp -------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "GDBRemoteClientBase.h"
 
 #include "llvm/ADT/StringExtras.h"
 
+#include "lldb/Host/ThreadLauncher.h"
 #include "lldb/Target/UnixSignals.h"
+#include "lldb/Utility/Connection.h"
+#include "lldb/Utility/Event.h"
 #include "lldb/Utility/LLDBAssert.h"
 
 #include "ProcessGDBRemoteLog.h"
 
 using namespace lldb;
 using namespace lldb_private;
 using namespace lldb_private::process_gdb_remote;
 using namespace std::chrono;
GDBRemoteClientBase::SendPacketAndReceiveResponseWithOutputSupport(
 
   return ReadPacketWithOutputSupport(response, m_comm.GetPacketTimeout(), true,
                                      output_callback);
 }
 
 GDBRemoteCommunication::PacketResult
 GDBRemoteClientBase::SendPacketAndWaitForResponseNoLock(
     llvm::StringRef payload, StringExtractorGDBRemote &response) {
-  PacketResult packet_result = SendPacketNoLock(payload);
-  if (packet_result != PacketResult::Success)
-    return packet_result;
-
+  PacketResult packet_result;
   const size_t max_response_retries = 3;
-  for (size_t i = 0; i < max_response_retries; ++i) {
-    packet_result = ReadPacket(response, m_comm.GetPacketTimeout(), true);
-    // Make sure we received a response
+  size_t response_retry = 0;
+
+  // TODO: timeout
+  if (!RequestComm(
+          [this, payload, &packet_result](MainLoopBase &) {
+            packet_result = m_comm.SendPacketNoLock(payload);
             if (packet_result != PacketResult::Success)
-      return packet_result;
+              BroadcastEvent(eBroadcastBitCommDone);
+          },
+          [this, &packet_result, &payload, &response,
+           &response_retry](MainLoopBase &) {
+            while (!m_comm_sync_packet_queue.empty()) {
+              response.Reset(m_comm_sync_packet_queue.front());
+              m_comm_sync_packet_queue.pop_front();
+
               // Make sure our response is valid for the payload that was sent
               if (response.ValidateResponse())
-      return packet_result;
+                break;
+
               // Response says it wasn't valid
               Log *log = GetLog(GDBRLog::Packets);
-    LLDB_LOGF(
-        log,
-        "error: packet with payload \"%.*s\" got invalid response \"%s\": %s",
-        int(payload.size()), payload.data(), response.GetStringRef().data(),
-        (i == (max_response_retries - 1))
+              LLDB_LOGF(log,
+                        "error: packet with payload \"%.*s\" got invalid "
+                        "response \"%s\": %s",
+                        int(payload.size()), payload.data(),
+                        response.GetStringRef().data(),
+                        (response_retry == (max_response_retries - 1))
                             ? "using invalid response and giving up"
                             : "ignoring response and waiting for another");
+
+              // If we get (max_response_retries) invalid responses,
+              // return the invalid response.
+              ++response_retry;
+              if (response_retry == max_response_retries)
+                break;
             }
+
+            packet_result = PacketResult::Success;
+            BroadcastEvent(eBroadcastBitCommDone);
+          }))
+    return PacketResult::ErrorDisconnected; // TODO
+
   return packet_result;
 }
 
 bool GDBRemoteClientBase::ShouldStop(const UnixSignals &signals,
                                      StringExtractorGDBRemote &response) {
   std::lock_guard<std::mutex> lock(m_mutex);
 
   if (m_async_count == 0)
if (!m_acquired)
     return;
   {
     std::unique_lock<std::mutex> lock(m_comm.m_mutex);
     --m_comm.m_async_count;
   }
   m_comm.m_cv.notify_one();
 }
 
+bool GDBRemoteClientBase::RequestComm(const MainLoop::Callback &send_callback,
+                                      const MainLoop::Callback &recv_callback) {
+  assert(m_comm_thread.IsJoinable());
+  ListenerSP listener_sp(Listener::MakeListener("gdb-remote.packet-sent"));
+  if (listener_sp->StartListeningForEvents(
+          this, eBroadcastBitCommDone | eBroadcastBitCommThreadExited |
+                    eBroadcastBitCommPacketRecv)) {
+    bool packet_read = false;
+    m_comm_loop_up->AddPendingCallback(send_callback);
+    // If the server replies very fast, the client may read response as part of
+    // ack packet.  Queue a read handler run to verify that we do not have any
+    // packets in the read buffer already.
+    m_comm_loop_up->AddPendingCallback(
+        [this, &packet_read, &recv_callback](MainLoopBase &loop) {
+          CommThreadReadHandler(loop);
+          if (recv_callback) {
+            if (!packet_read && !m_comm_sync_packet_queue.empty()) {
+              recv_callback(loop);
+              packet_read = true;
+            }
+          }
+        });
+
+    EventSP event_sp;
+    while (!m_comm_thread_exited &&
+           listener_sp->GetEvent(event_sp, llvm::None)) {
+      if (event_sp->GetType() & eBroadcastBitCommDone)
+        return true;
+      if (event_sp->GetType() & eBroadcastBitCommThreadExited)
+        return false;
+      if (event_sp->GetType() & eBroadcastBitCommPacketRecv) {
+        m_comm_loop_up->AddPendingCallback(
+            [this, &packet_read, &recv_callback](MainLoopBase &loop) {
+              if (!packet_read && !m_comm_sync_packet_queue.empty()) {
+                recv_callback(loop);
+                packet_read = true;
+              }
+            });
+      }
+    }
+  }
+  return false;
+}
+
 GDBRemoteClientBase::PacketResult
 GDBRemoteClientBase::ReadPacket(StringExtractorGDBRemote &response,
                                 Timeout<std::micro> timeout,
                                 bool sync_on_timeout) {
-  return m_comm.ReadPacket(response, timeout, sync_on_timeout);
+  PacketResult packet_result;
+
+  if (!RequestComm([](MainLoopBase &) {},
+                   [this, &packet_result, &response](MainLoopBase &) {
+                     assert(!m_comm_sync_packet_queue.empty());
+
+                     response.Reset(m_comm_sync_packet_queue.front());
+                     m_comm_sync_packet_queue.pop_front();
+
+                     packet_result = PacketResult::Success;
+                     BroadcastEvent(eBroadcastBitCommDone);
+                   }))
+    return PacketResult::ErrorDisconnected; // TODO
+
+  return packet_result;
 }
 
 GDBRemoteClientBase::PacketResult
 GDBRemoteClientBase::SendPacketNoLock(llvm::StringRef payload) {
-  return m_comm.SendPacketNoLock(payload);
+  PacketResult packet_result;
+
+  if (!RequestComm(
+          [this, &packet_result, payload](MainLoopBase &) {
+            packet_result = m_comm.SendPacketNoLock(payload);
+            BroadcastEvent(eBroadcastBitCommDone);
+          },
+          nullptr))
+    return PacketResult::ErrorDisconnected; // TODO
+
+  return packet_result;
 }
 
-size_t GDBRemoteClientBase::SendAck() { return m_comm.SendAck(); }
+size_t GDBRemoteClientBase::SendAck() {
+  size_t bytes_written;
+
+  if (RequestComm(
+          [this, &bytes_written](MainLoopBase &) {
+            bytes_written = m_comm.SendAck();
+            BroadcastEvent(eBroadcastBitCommDone);
+          },
+          nullptr))
+    return bytes_written;
+  return 0;
+}
 
 bool GDBRemoteClientBase::SendCtrlC() {
-  const char ctrl_c = '\x03';
   ConnectionStatus status = eConnectionStatusSuccess;
-  size_t bytes_written = m_comm.Write(&ctrl_c, 1, status, nullptr);
+  size_t bytes_written;
+
+  if (RequestComm(
+          [this, &bytes_written, &status](MainLoopBase &) {
+            const char ctrl_c = '\x03';
+            bytes_written = m_comm.Write(&ctrl_c, 1, status, nullptr);
+            BroadcastEvent(eBroadcastBitCommDone);
+          },
+          nullptr))
     return bytes_written != 0;
+  return false;
 }
 
 bool GDBRemoteClientBase::IsConnected() const { return m_comm.IsConnected(); }
 
 ConnectionStatus GDBRemoteClientBase::Disconnect(Status *error_ptr) {
+  StopThread();
   return m_comm.Disconnect(error_ptr);
 }
+
+llvm::Error GDBRemoteClientBase::StartThread() {
+  Log *log = GetLog(GDBRLog::Process);
+  LLDB_LOG(log, "Starting comm thread");
+  assert(!m_comm_thread.IsJoinable());
+
+  // Instantiate the loop early to avoid races.
+  m_comm_loop_up.reset(new MainLoop());
+  m_comm_thread_exited = false;
+
+  auto maybe_thread = ThreadLauncher::LaunchThread(
+      "<lldb.gdb-remote.comm>", [this] { return CommThread(); });
+  if (!maybe_thread)
+    return maybe_thread.takeError();
+
+  m_comm_thread = *maybe_thread;
+  assert(m_comm_thread.IsJoinable());
+
+  return llvm::Error::success();
+}
+
+void GDBRemoteClientBase::StopThread() {
+  Log *log = GetLog(GDBRLog::Process);
+
+  if (!m_comm_thread.IsJoinable())
+    return;
+
+  LLDB_LOG(log, "Stopping comm thread");
+  m_comm_loop_up->AddPendingCallback(
+      [](MainLoopBase &loop) { loop.RequestTermination(); });
+  if (m_comm_thread.Join(nullptr).Success())
+    m_comm_loop_up.reset(nullptr);
+  assert(m_comm_thread_exited);
+}
+
+void GDBRemoteClientBase::CommThreadReadHandler(MainLoopBase &loop) {
+  StringExtractorGDBRemote response;
+
+  while (true) {
+    // Read incoming packets until we reach timeout (i.e. all pending packets have been processed).
+    PacketResult packet_result =
+        m_comm.ReadPacket(response, std::chrono::seconds(0), false);
+
+    if (packet_result == PacketResult::ErrorReplyTimeout)
+      break;
+    if (packet_result != PacketResult::Success) {
+      loop.RequestTermination();
+      break;
+    }
+
+    m_comm_sync_packet_queue.push_back(response.GetStringRef().str());
+    BroadcastEvent(eBroadcastBitCommPacketRecv);
+  }
+}
+
+lldb::thread_result_t GDBRemoteClientBase::CommThread() {
+  Log *log = GetLog(GDBRLog::Process);
+  assert(m_comm_loop_up);
+
+  Status error;
+  auto handle = m_comm_loop_up->RegisterReadObject(
+      m_comm.GetConnection()->GetReadObject(),
+      [this](MainLoopBase &loop) { CommThreadReadHandler(loop); }, error);
+  if (error.Success())
+    error = m_comm_loop_up->Run();
+  if (!error.Success()) {
+    // TODO
+  }
+
+  LLDB_LOG(log, "Comm thread exiting");
+  m_comm_thread_exited = true;
+  BroadcastEvent(eBroadcastBitCommThreadExited);
+
+  return {};
+}

lldb/source/Plugins/Process/gdb-remote/ProcessGDBRemote.cpp

Status ProcessGDBRemote::ConnectToDebugserver(llvm::StringRef connect_url) {
   }
 
   if (!m_gdb_comm.IsConnected()) {
     if (error.Success())
       error.SetErrorString("not connected to remote gdb server");
     return error;
   }
 
+  llvm::Error thread_error = m_gdb_comm.StartThread();
+  if (thread_error) {
+    m_gdb_comm.Disconnect();
+    return Status(std::move(thread_error));
+  }
   // We always seem to be able to open a connection to a local port so we need
   // to make sure we can then send data to it. If we can't then we aren't
   // actually connected to anything, so try and do the handshake with the
   // remote GDB server and make sure that goes alright.
   if (!m_gdb_comm.HandshakeWithServer(&error)) {
     m_gdb_comm.Disconnect();
     if (error.Success())
       error.SetErrorString("not connected to remote gdb server");