PilotAware-OGN hangs ocasionally, but ogn-rf and ogn-decode keep running fine

[Kevin W]

Hi

I have had a couple of times now when port 80 stops responding, but port 8080 and 8081 are still responding.  Digging into it, port 50000 and 50001 are still streaming their output fine, but port 50002 has stopped - nothing going past.  A quick ^X to get procServ to restart PilotAware-OGN on 50002 are we are quickly back to normal.

Anything we can do to get you more info on what goes wrong at that point?  A more recent version of procServ would enable the --history option so we could see the last things printed?

Cheers
Kev

[Admin]

Hi Kev

Can you see if the process is still running

[Kevin W]

Hi Lee

Im pretty sure the process was still running, otherwise procServ would have restarted it? I didnt check though - next time

Cheers
Kev

[Admin]

Hi Kev

Ok, what would be useful is to run strace on the process
First get the pid

ps -eaf | grep PilotAware
sudo strace -p <pid>

Thx
Lee

[Kevin W]

A slightly different failure mode today,  Pilotaware-OGN stopped responding on port 80 at 04:22 this morning, but seems to still be working other than the web server, although messages on 50002 looked very different between before and after the restart.  Oh and before the restart it had spawned ~100 instances of the PA_TxSBS1 thread!

pi@ognpaw:~ $ telnet localhost 80
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
GET / HTTP/1.1
Host: ognpaw.unseen.org

[nothing]

pi@ognpaw:~ $ netcat localhost 50002
@@@ Welcome to procServ (procServ Process Server 2.6.0)
@@@ Child "./PilotAware-OGN.exe" started at: Sat Apr 13 11:17:09 2019
(08:31:20) MLAT-SRVR : STATUS 400CEE Latency=(08:31:19.202) latency=1 error=114
(PARSE-FAIL)HTTP_Server.Exec() ... Client from 10.10.10.8:48536
(PARSE-FAIL)HTTP_Server.Exec() ... Request for /
(08:31:21) MLAT-SRVR : STATUS AB00DD Latency=(08:31:19.651) latency=1 error=132
(08:31:21) MLAT-SRVR : STATUS 401DC6 Latency=(08:31:20.931) latency=0 error=113
(08:31:22) MLAT-SRVR : STATUS AB00DD Latency=(08:31:20.251) latency=2 error=126
(08:31:23) COMMON    : LOCK: Blocked=0.00876    FEEDOUT(8) ->    FEEDOUT
(08:31:23) COMMON    : LOCK: Blocked=0.00892    FEEDOUT(8) ->    FEEDOUT
(08:31:23) COMMON    : LOCK: Blocked=0.00902    FEEDOUT(8) ->    FEEDOUT


pi@ognpaw:~ $ netcat localhost 50002
@@@ procServ server started at: Sat Apr 13 11:17:09 2019
@@@ Child "./PilotAware-OGN.exe" started at: Sat Apr 13 11:17:09 2019
(PARSE-FAIL)HTTP_Server.Exec() ... Client from 10.10.10.8:48590
(PARSE-FAIL)HTTP_Server.Exec() ... Request for /
(08:32:27) TRX-RF    : BCN-TX icao=FD4054 acft=1f lat=52.32267 lon=-0.70617 alt=92
PAWRT,40626E,TIMEOUT
(08:32:28) MLAT-SRVR : STATUS 400796 Latency=(08:32:26.522) latency=1 error=35
(08:32:29) MLAT-SRVR : STATUS 400796 Latency=(08:32:26.770) latency=2 error=117
(08:32:29) MLAT-SRVR : STATUS 400796 Latency=(08:32:27.490) latency=2 error=86
(08:32:29) MLAT-SRVR : STATUS 400CEE Latency=(08:32:26.876) latency=2 error=106
(08:32:29) MLAT-SRVR : STATUS 401DC6 Latency=(08:32:26.330) latency=3 error=241
(08:32:29) MLAT-SRVR : STATUS 400CEE Latency=(08:32:27.428) latency=2 error=111
(08:32:29) MLAT-SRVR : STATUS 400796 Time Reverse (Abort Update) 08:32:27 to 08:32:26
(08:32:30) INTF-OGN  : APRS_SERVER: (recv) # aprsc 2.1.4-g408ed49 27 May 2019 08:32:30 GMT GLIDERN3 85.188.1.173:14580
(08:32:30) MLAT-SRVR : STATUS 400796 Latency=(08:32:27.829) latency=2 error=108
(08:32:30) MLAT-SRVR : STATUS 400CEE Latency=(08:32:30.402) latency=0 error=131
(PARSE-FAIL)HTTP_Server.Exec() ... Client from 10.10.10.8:48594
(08:32:32) MLAT-SRVR : STATUS 400CEE Latency=(08:32:31.639) latency=0 error=95
(08:32:32) COMMON    : LOCK: Blocked=0.00928    FEEDOUT(23) ->    FEEDOUT
(08:32:32) COMMON    : LOCK: Blocked=0.00939    FEEDOUT(24) ->    FEEDOUT
(08:32:32) COMMON    : LOCK: Blocked=0.01185    FEEDOUT(27) ->    FEEDOUT
(08:32:32) COMMON    : LOCK: Blocked=0.00688    FEEDOUT(28) ->    FEEDOUT
(08:32:32) COMMON    : LOCK: Blocked=0.00716    FEEDOUT(28) ->    FEEDOUT
(08:32:32) COMMON    : LOCK: Blocked=0.00724    FEEDOUT(28) ->    FEEDOUT
(08:32:32) COMMON    : LOCK: Blocked=0.00729    FEEDOUT(28) ->    FEEDOUT
(PARSE-FAIL)HTTP_Server.Exec() ... Request for /
(08:32:32) MLAT-SRVR : STATUS 400796 Latency=(08:32:30.950) latency=1 error=68
(08:32:33) MLAT-SRVR : STATUS 401DC6 Latency=(08:32:30.551) latency=2 error=217


pi@ognpaw:~ $ ps -auxwww | grep PilotAware-OGN
root       789  1.1  8.3 1016376 79052 pts/3   Ssl+ May26  13:10 ./PilotAware-OGN.exe

pi@ognpaw:~ $ sudo strace -p 789
Process 789 attached
futex(0x765174c8, FUTEX_WAIT, 800, NULL

pi@ognpaw:~ $ ps -T -p 789
 PID  SPID TTY          TIME CMD
  789   789 pts/3    00:00:00 PilotAware-OGN.
  789   794 pts/3    00:00:09 PA_httpd
  789   800 pts/3    00:00:03 OGN_recv
  789   802 pts/3    00:00:36 OGN_send
  789   803 pts/3    00:04:25 PA_RxTxRadio
  789   804 pts/3    00:00:00 PA_TxSBS1
  789   805 pts/3    00:00:23 PA_ParseSBS1
  789   806 pts/3    00:00:00 aprs_local
  789   807 pts/3    00:01:09 PA_SysMonitor
  789 11651 pts/3    00:00:01 PA_TxSBS1
  789 11667 pts/3    00:00:01 PA_TxSBS1
  789 11679 pts/3    00:00:01 PA_TxSBS1
  789 11721 pts/3    00:00:01 PA_TxSBS1
  789 11736 pts/3    00:00:01 PA_TxSBS1
  789 11753 pts/3    00:00:01 PA_TxSBS1
  789 11759 pts/3    00:00:01 PA_TxSBS1
[~100 ocourances of  PA_TxSBS1]


pi@ognpaw:~ $ sudo strace -p 794
Process 794 attached
nanosleep({0, 313648546}, 0x76d13da4)   = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [SEGV], 8) = 0
rt_sigaction(SIGCHLD, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigprocmask(SIG_SETMASK, [SEGV], NULL, 8) = 0
nanosleep({1, 0}, 0x76d13da4)           = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [SEGV], 8) = 0
rt_sigaction(SIGCHLD, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigprocmask(SIG_SETMASK, [SEGV], NULL, 8) = 0


pi@ognpaw:~ $ sudo strace -p 794
Process 794 attached
nanosleep({0, 313648546}, 0x76d13da4)   = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [SEGV], 8) = 0
rt_sigaction(SIGCHLD, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigprocmask(SIG_SETMASK, [SEGV], NULL, 8) = 0
nanosleep({1, 0}, 0x76d13da4)           = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [SEGV], 8) = 0
rt_sigaction(SIGCHLD, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigprocmask(SIG_SETMASK, [SEGV], NULL, 8) = 0


pi@ognpaw:~ $ sudo strace -p 794
Process 794 attached
nanosleep({0, 313648546}, 0x76d13da4)   = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [SEGV], 8) = 0
rt_sigaction(SIGCHLD, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigprocmask(SIG_SETMASK, [SEGV], NULL, 8) = 0
nanosleep({1, 0}, 0x76d13da4)           = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [SEGV], 8) = 0
rt_sigaction(SIGCHLD, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigprocmask(SIG_SETMASK, [SEGV], NULL, 8) = 0


pi@ognpaw:~ $ sudo strace -p 800
Process 800 attached
read(7, "HTTP_Server.Exec() ... Client fr"..., 1023) = 53
write(1, "(PARSE-FAIL)HTTP_Server.Exec() ."..., 64) = 64
nanosleep({0, 100000000}, NULL)         = 0
read(7, "HTTP_Server.Exec() ... Request f"..., 1023) = 38
write(1, "(PARSE-FAIL)HTTP_Server.Exec() ."..., 49) = 49
nanosleep({0, 100000000}, NULL)         = 0


pi@ognpaw:~ $ sudo strace -p 802
Process 802 attached
select(26, [25], [], NULL, {0, 5565})   = 0 (Timeout)
gettimeofday({1558945560, 952377}, NULL) = 0
clock_gettime(CLOCK_MONOTONIC, {68343, 542101910}) = 0
clock_gettime(CLOCK_MONOTONIC, {68343, 542165816}) = 0
select(26, [25], [], NULL, {0, 100000}) = 0 (Timeout)
gettimeofday({1558945561, 53272}, NULL) = 0
clock_gettime(CLOCK_MONOTONIC, {68343, 643005896}) = 0
clock_gettime(CLOCK_MONOTONIC, {68343, 643069385}) = 0

pi@ognpaw:~ $ sudo strace -p 803
Process 803 attached
nanosleep({0, 10000000}, NULL)          = 0
read(12, 0x753fea18, 1024)              = -1 EAGAIN (Resource temporarily unavailable)
gettimeofday({1558945602, 115036}, {0, 0}) = 0
nanosleep({0, 10000000}, NULL)          = 0
read(12, 0x753fea18, 1024)              = -1 EAGAIN (Resource temporarily unavailable)
gettimeofday({1558945602, 125400}, {0, 0}) = 0


pi@ognpaw:~ $ sudo strace -p 804
Process 804 attached
accept(9,


pi@ognpaw:~ $ sudo strace -p 805
Process 805 attached
select(25, [24], NULL, NULL, {9, 769852}) = 1 (in [24], left {9, 560389})
gettimeofday({1558945658, 434729}, NULL) = 0
read(24, "MSG,3,1,1,40650B,1,2019/05/27,08"..., 1024) = 123
gettimeofday({1558945658, 435286}, NULL) = 0
clock_gettime(CLOCK_MONOTONIC, {68441, 25215299}) = 0
clock_gettime(CLOCK_MONOTONIC, {68441, 25293216}) = 0
gettimeofday({1558945658, 435918}, NULL) = 0
stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=118, ...}) = 0
stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=118, ...}) = 0
gettimeofday({1558945658, 436749}, NULL) = 0
gettimeofday({1558945658, 436940}, NULL) = 0
stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=118, ...}) = 0
write(2, "(08:27:38) MLAT-SRVR : ", 23) = 23
write(2, "STATUS 40650B Latency=(08:27:35."..., 56) = 56
gettimeofday({1558945658, 438028}, NULL) = 0
gettimeofday({1558945658, 438255}, NULL) = 0
gettimeofday({1558945658, 438486}, NULL) = 0
gettimeofday({1558945658, 438656}, NULL) = 0


pi@ognpaw:~ $ sudo strace -p 806
Process 806 attached
accept(8,


pi@ognpaw:~ $ sudo strace -p 807
Process 807 attached
nanosleep({0, 294656457}, 0x733fe79c)   = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [SEGV], 8) = 0
rt_sigaction(SIGCHLD, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigprocmask(SIG_SETMASK, [SEGV], NULL, 8) = 0
nanosleep({1, 0}, 0x733fe79c)           = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [SEGV], 8) = 0
rt_sigaction(SIGCHLD, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigprocmask(SIG_SETMASK, [SEGV], NULL, 8) = 0


pi@ognpaw:~ $ sudo strace -p 11651
Process 11651 attached
nanosleep({1, 0}, NULL)                 = 0
gettimeofday({1558945739, 270574}, NULL) = 0
stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=118, ...}) = 0
clock_gettime(CLOCK_MONOTONIC, {68521, 860491088}) = 0
clock_gettime(CLOCK_MONOTONIC, {68521, 860559838}) = 0
gettimeofday({1558945739, 270996}, NULL) = 0
nanosleep({1, 0}, NULL)                 = 0
gettimeofday({1558945740, 271381}, NULL) = 0
stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=118, ...}) = 0
clock_gettime(CLOCK_MONOTONIC, {68522, 861522577}) = 0
clock_gettime(CLOCK_MONOTONIC, {68522, 861652576}) = 0
gettimeofday({1558945740, 272181}, NULL) = 0
nanosleep({1, 0}, ^CProcess 11651 detached


Restarting the service with ^X it is fine again:

i@ognpaw:~ $ telnet localhost 80
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
GET / HTTP/1.1
Host: ognpaw.unseen.org

HTTP/1.1 200 OK
Connection: close
Server: PilotAwarehttpd/0.1.0
Content-Length: 4833
Content-Type: text/html

<html><title><Status>[etc]


pi@ognpaw:~ $ netcat localhost 50002
@@@ procServ server started at: Sat Apr 13 11:17:09 2019
@@@ Child "./PilotAware-OGN.exe" started at: Mon May 27 08:34:24 2019
(PARSE-FAIL)HTTP_Server.Exec() ... Client from 10.10.10.8:49290
(PARSE-FAIL)HTTP_Server.Exec() ... Request for /
(08:38:34) MLAT-SRVR : STATUS 400F6F Latency=(08:38:33.115) latency=1 error=104
(08:38:34) TRX-RF    : BCN-TX icao=FD4054 acft=1f lat=52.32267 lon=-0.70617 alt=92
(08:38:34) MLAT-SRVR : STATUS 401906 Latency=(08:38:32.819) latency=1 error=123
(08:38:34) MLAT-SRVR : STATUS 400CEE Latency=(08:38:32.442) latency=2 error=141
(08:38:35) MLAT-SRVR : STATUS 4035E1 Latency=(08:38:33.657) latency=1 error=181
(08:38:35) MLAT-SRVR : STATUS 40543F Latency=(08:38:34.490) latency=1 error=178
(08:38:35) MLAT-SRVR : STATUS 400CEE Latency=(08:38:34.363) latency=1 error=93
(08:38:36) INTF-OGN  : APRS_SERVER: (recv) # aprsc 2.1.4-g408ed49 27 May 2019 08:38:36 GMT GLIDERN5 88.99.111.134:14580
(08:38:36) MLAT-SRVR : STATUS 401906 Latency=(08:38:35.555) latency=0 error=119
(08:38:37) MLAT-SRVR : STATUS 406FA5 Latency=(08:38:34.825) latency=2 error=410
(08:38:37) MLAT-SRVR : STATUS 40543F Latency=(08:38:36.578) latency=0 error=128
(08:38:38) MLAT-SRVR : STATUS 4076BF Latency=(08:38:35.102) latency=3 error=161
(08:38:38) MLAT-SRVR : STATUS 400CEE Latency=(08:38:37.413) latency=1 error=166
(08:38:38) MLAT-SRVR : STATUS 400CEE Latency=(08:38:38.103) latency=0 error=146
(PARSE-FAIL)HTTP_Server.Exec() ... Client from 10.10.10.8:49302
(PARSE-FAIL)HTTP_Server.Exec() ... Request for /

pi@ognpaw:~ $ ps -T -p 21293
  PID  SPID TTY          TIME CMD
21293 21293 pts/3    00:00:00 PilotAware-OGN.
21293 21298 pts/3    00:00:00 PA_httpd
21293 21299 pts/3    00:00:00 OGN_recv
21293 21300 pts/3    00:00:00 OGN_send
21293 21301 pts/3    00:00:01 PA_RxTxRadio
21293 21303 pts/3    00:00:00 PA_TxSBS1
21293 21304 pts/3    00:00:03 PA_ParseSBS1
21293 21305 pts/3    00:00:00 aprs_local
21293 21306 pts/3    00:00:00 PA_SysMonitor
21293 21639 pts/3    00:00:00 PA_TxSBS1


Cheers
Kev

[Ian Melville]

Kevin, can I suggest you change your post to not use Smileys.

Click o the plus sign for "Attachments and other options", and you will see the option.

[Admin]

Hi Kev

This is useful info
Are you connecting any remote processes over sockets to ognr

This looks like connections to tcp 30103
maybe running a local virtual radar ?

Have you opened ports on the internet which are being accessed ?

*edit*
so further update upon investigation
There is a listening TCP socket on port 30103, this serves SBS based data (similar to dump1090 output)
if you have hundreds of these processes running (actually threads not processes), this means there are incoming TCP connections, one for each thread.
So the question is where are all these TCP requests coming from ?

This will probably grind the system to a halt, or hit a max resource limit (max file descriptors for example)

Thx
Lee

[Kevin W]

Hi Lee

Thanks very much for the investigations. 

The only thing 'different' on my setup is the FR24 feed, and the only things that connects to are localhost:30002 (dump1090) for the feed in.

Only ports open to the outside world are the web servers.

The 30103 connections all appear to be internal, from the VirtualRadar server:

Code: [Select]

root@ognpaw:/home/pi# lsof | grep 30103
PilotAwar  1281              root    8u     IPv4      13481      0t0        TCP *:30103 (LISTEN)
PilotAwar  1281              root  247u     IPv4    9111476      0t0        TCP localhost:30103->localhost:55434 (CLOSE_WAIT)
PilotAwar  1281              root  249u     IPv4    9199631      0t0        TCP localhost:30103->localhost:55436 (ESTABLISHED)
PA_httpd   1281  1286        root    8u     IPv4      13481      0t0        TCP *:30103 (LISTEN)
PA_httpd   1281  1286        root  247u     IPv4    9111476      0t0        TCP localhost:30103->localhost:55434 (CLOSE_WAIT)
PA_httpd   1281  1286        root  249u     IPv4    9199631      0t0        TCP localhost:30103->localhost:55436 (ESTABLISHED)
OGN_recv   1281  1290        root    8u     IPv4      13481      0t0        TCP *:30103 (LISTEN)
OGN_recv   1281  1290        root  247u     IPv4    9111476      0t0        TCP localhost:30103->localhost:55434 (CLOSE_WAIT)
OGN_recv   1281  1290        root  249u     IPv4    9199631      0t0        TCP localhost:30103->localhost:55436 (ESTABLISHED)
OGN_send   1281  1291        root    8u     IPv4      13481      0t0        TCP *:30103 (LISTEN)
OGN_send   1281  1291        root  247u     IPv4    9111476      0t0        TCP localhost:30103->localhost:55434 (CLOSE_WAIT)
OGN_send   1281  1291        root  249u     IPv4    9199631      0t0        TCP localhost:30103->localhost:55436 (ESTABLISHED)
PA_RxTxRa  1281  1292        root    8u     IPv4      13481      0t0        TCP *:30103 (LISTEN)
PA_RxTxRa  1281  1292        root  247u     IPv4    9111476      0t0        TCP localhost:30103->localhost:55434 (CLOSE_WAIT)
PA_RxTxRa  1281  1292        root  249u     IPv4    9199631      0t0        TCP localhost:30103->localhost:55436 (ESTABLISHED)
PA_TxSBS1  1281  1293        root    8u     IPv4      13481      0t0        TCP *:30103 (LISTEN)
PA_TxSBS1  1281  1293        root  247u     IPv4    9111476      0t0        TCP localhost:30103->localhost:55434 (CLOSE_WAIT)
PA_TxSBS1  1281  1293        root  249u     IPv4    9199631      0t0        TCP localhost:30103->localhost:55436 (ESTABLISHED)
PA_ParseS  1281  1295        root    8u     IPv4      13481      0t0        TCP *:30103 (LISTEN)
PA_ParseS  1281  1295        root  247u     IPv4    9111476      0t0        TCP localhost:30103->localhost:55434 (CLOSE_WAIT)
PA_ParseS  1281  1295        root  249u     IPv4    9199631      0t0        TCP localhost:30103->localhost:55436 (ESTABLISHED)
aprs_loca  1281  1296        root    8u     IPv4      13481      0t0        TCP *:30103 (LISTEN)
aprs_loca  1281  1296        root  247u     IPv4    9111476      0t0        TCP localhost:30103->localhost:55434 (CLOSE_WAIT)
aprs_loca  1281  1296        root  249u     IPv4    9199631      0t0        TCP localhost:30103->localhost:55436 (ESTABLISHED)
PA_SysMon  1281  1297        root    8u     IPv4      13481      0t0        TCP *:30103 (LISTEN)
PA_SysMon  1281  1297        root  247u     IPv4    9111476      0t0        TCP localhost:30103->localhost:55434 (CLOSE_WAIT)
PA_SysMon  1281  1297        root  249u     IPv4    9199631      0t0        TCP localhost:30103->localhost:55436 (ESTABLISHED)
PA_TxSBS1  1281 18672        root    8u     IPv4      13481      0t0        TCP *:30103 (LISTEN)
PA_TxSBS1  1281 18672        root  247u     IPv4    9111476      0t0        TCP localhost:30103->localhost:55434 (CLOSE_WAIT)
PA_TxSBS1  1281 18672        root  249u     IPv4    9199631      0t0        TCP localhost:30103->localhost:55436 (ESTABLISHED)
PA_TxSBS1  1281 19316        root    8u     IPv4      13481      0t0        TCP *:30103 (LISTEN)
PA_TxSBS1  1281 19316        root  247u     IPv4    9111476      0t0        TCP localhost:30103->localhost:55434 (CLOSE_WAIT)
PA_TxSBS1  1281 19316        root  249u     IPv4    9199631      0t0        TCP localhost:30103->localhost:55436 (ESTABLISHED)
PA_TxSBS1  1281 19566        root    8u     IPv4      13481      0t0        TCP *:30103 (LISTEN)
PA_TxSBS1  1281 19566        root  247u     IPv4    9111476      0t0        TCP localhost:30103->localhost:55434 (CLOSE_WAIT)
PA_TxSBS1  1281 19566        root  249u     IPv4    9199631      0t0        TCP localhost:30103->localhost:55436 (ESTABLISHED)
PA_TxSBS1  1281 19750        root    8u     IPv4      13481      0t0        TCP *:30103 (LISTEN)
PA_TxSBS1  1281 19750        root  247u     IPv4    9111476      0t0        TCP localhost:30103->localhost:55434 (CLOSE_WAIT)
PA_TxSBS1  1281 19750        root  249u     IPv4    9199631      0t0        TCP localhost:30103->localhost:55436 (ESTABLISHED)
PA_TxSBS1  1281 19924        root    8u     IPv4      13481      0t0        TCP *:30103 (LISTEN)
PA_TxSBS1  1281 19924        root  247u     IPv4    9111476      0t0        TCP localhost:30103->localhost:55434 (CLOSE_WAIT)
PA_TxSBS1  1281 19924        root  249u     IPv4    9199631      0t0        TCP localhost:30103->localhost:55436 (ESTABLISHED)
mono       1403                pi    5u     IPv4    9217526      0t0        TCP localhost:55436->localhost:30103 (ESTABLISHED)
SGen       1403  1404          pi    5u     IPv4    9217526      0t0        TCP localhost:55436->localhost:30103 (ESTABLISHED)
Finalizer  1403  1405          pi    5u     IPv4    9217526      0t0        TCP localhost:55436->localhost:30103 (ESTABLISHED)
Timer-Sch  1403  1422          pi    5u     IPv4    9217526      0t0        TCP localhost:55436->localhost:30103 (ESTABLISHED)
MessagePr  1403  1481          pi    5u     IPv4    9217526      0t0        TCP localhost:55436->localhost:30103 (ESTABLISHED)
MessagePr  1403  1486          pi    5u     IPv4    9217526      0t0        TCP localhost:55436->localhost:30103 (ESTABLISHED)
MergedFee  1403  1488          pi    5u     IPv4    9217526      0t0        TCP localhost:55436->localhost:30103 (ESTABLISHED)
Thread     1403  1493          pi    5u     IPv4    9217526      0t0        TCP localhost:55436->localhost:30103 (ESTABLISHED)
Connectio  1403  1500          pi    5u     IPv4    9217526      0t0        TCP localhost:55436->localhost:30103 (ESTABLISHED)
PictureLo  1403  1504          pi    5u     IPv4    9217526      0t0        TCP localhost:55436->localhost:30103 (ESTABLISHED)
Thread     1403  1527          pi    5u     IPv4    9217526      0t0        TCP localhost:55436->localhost:30103 (ESTABLISHED)
Thread     1403  1528          pi    5u     IPv4    9217526      0t0        TCP localhost:55436->localhost:30103 (ESTABLISHED)
Thread     1403  1529          pi    5u     IPv4    9217526      0t0        TCP localhost:55436->localhost:30103 (ESTABLISHED)
Thread     1403  1530          pi    5u     IPv4    9217526      0t0        TCP localhost:55436->localhost:30103 (ESTABLISHED)
Thread     1403  1749          pi    5u     IPv4    9217526      0t0        TCP localhost:55436->localhost:30103 (ESTABLISHED)
Thread     1403  2121          pi    5u     IPv4    9217526      0t0        TCP localhost:55436->localhost:30103 (ESTABLISHED)
Thread     1403 15429          pi    5u     IPv4    9217526      0t0        TCP localhost:55436->localhost:30103 (ESTABLISHED)
Timer-Sch  1403 19915          pi    5u     IPv4    9217526      0t0        TCP localhost:55436->localhost:30103 (ESTABLISHED)
Connectio  1403 19925          pi    5u     IPv4    9217526      0t0        TCP localhost:55436->localhost:30103 (ESTABLISHED)

root@ognpaw:/home/pi# ps -auxwww | grep 1403
pi        1403  8.5  9.2 146560 87040 pts/5    Sl+  May27  96:12 mono bin/VirtualRadar.exe -nogui
root     20014  0.0  0.2   4276  2016 pts/0    S+   12:34   0:00 grep 1403

root@ognpaw:/home/pi# ps -auxwww | grep 1281
root      1281  0.8  8.2 229944 78032 pts/3    Ssl+ May27   9:43 ./PilotAware-OGN.exe
root     20025  0.0  0.2   4276  2016 pts/0    S+   12:34   0:00 grep 1281

Only a few SBS running today:

Code: [Select]

root@ognpaw:/home/pi# ps -T -p 1281
  PID  SPID TTY          TIME CMD
 1281  1281 pts/3    00:00:00 PilotAware-OGN.
 1281  1286 pts/3    00:00:00 PA_httpd
 1281  1290 pts/3    00:00:02 OGN_recv
 1281  1291 pts/3    00:00:37 OGN_send
 1281  1292 pts/3    00:04:24 PA_RxTxRadio
 1281  1293 pts/3    00:00:00 PA_TxSBS1
 1281  1295 pts/3    00:00:28 PA_ParseSBS1
 1281  1296 pts/3    00:00:00 aprs_local
 1281  1297 pts/3    00:01:19 PA_SysMonitor
 1281 18672 pts/3    00:00:00 PA_TxSBS1
 1281 19316 pts/3    00:00:00 PA_TxSBS1
 1281 19566 pts/3    00:00:00 PA_TxSBS1
 1281 19750 pts/3    00:00:00 PA_TxSBS1
 1281 19924 pts/3    00:00:00 PA_TxSBS1
 1281 20118 pts/3    00:00:00 PA_TxSBS1
[this is the full list]

[Admin]

Hi Kevin,

So I wonder when connecting to Virtual Radar server, does that create another instance ?
If so, then each instance will connect to this port,
Durrhhhh actually that is it exactly

So how many incoming connections to the VRS (port 9000) are there ?

I presume each connection to <IP>:9000 will spawn a new connection to the internal port 30103

Only ports open to the outside world are the web servers.

can I ask which ports ?

Thx
Lee

[Kevin W]

Hi Lee

Sorry missed your previous reply. 

Only ports open to the outside world are 8080, 8081, and 80 (seen as 8082 from the outside world), to allow me to do up time monitoring.

PilotAware-OGN stopped responding again this afternoon.  Process still running:

Code: [Select]

root@ognpaw:/home/pi# ps -auxwww | grep PilotAware-OGN
pi        1464  0.0  0.2   4168  1980 ?        Ss   May28   1:28 /usr/bin/procServ -p /tmp/procServ-50002.pid -k ^X --killsig 15 -x ^C -i ^D -c /home/pi/rtlsdr-ogn 50002 ./PilotAware-OGN.exe
root      1469  0.9  8.3 238136 78756 pts/3    Ssl+ May28  57:51 ./PilotAware-OGN.exe
root     31069  0.0  0.2   4276  1912 pts/6    S+   13:37   0:00 grep PilotAware-OGN
Threads:

Code: [Select]

root@ognpaw:/home/pi# ps -T -p 1469
  PID  SPID TTY          TIME CMD
 1469  1469 pts/3    00:00:00 PilotAware-OGN.
 1469  1474 pts/3    00:00:09 PA_httpd
 1469  1479 pts/3    00:00:23 OGN_recv
 1469  1480 pts/3    00:03:39 OGN_send
 1469  1481 pts/3    00:25:09 PA_RxTxRadio
 1469  1482 pts/3    00:00:00 PA_TxSBS1
 1469  1483 pts/3    00:03:53 PA_ParseSBS1
 1469  1484 pts/3    00:00:00 aprs_local
 1469  1485 pts/3    00:07:37 PA_SysMonitor
 1469 20463 pts/3    00:00:26 PA_TxSBS1
 1469 30664 pts/3    00:00:00 PA_httpd_acc
 1469 30711 pts/3    00:00:00 PA_TxSBS1
 1469 30786 pts/3    00:00:00 PA_TxSBS1

Code: [Select]

root@ognpaw:/home/pi# ps -T -p 1469 | awk '{print $2}' | grep -v SPID | xargs -L 1 timeout 2 strace -p
Process 1469 attached
futex(0x7645b4c8, FUTEX_WAIT, 1479, NULLProcess 1469 detached
 <detached ...>
Process 1474 attached
futex(0x69aff4c8, FUTEX_WAIT, 30664, NULLProcess 1474 detached
 <detached ...>
Process 1479 attached
futex(0xd31bc4, FUTEX_WAIT_PRIVATE, 2, NULLProcess 1479 detached
 <detached ...>
Process 1480 attached
nanosleep({0, 255934380}, 0x75c59ecc)   = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [SEGV], 8) = 0
rt_sigaction(SIGCHLD, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigprocmask(SIG_SETMASK, [SEGV], NULL, 8) = 0
nanosleep({1, 0}, 0x75c59ecc)           = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [SEGV], 8) = 0
rt_sigaction(SIGCHLD, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigprocmask(SIG_SETMASK, [SEGV], NULL, 8) = 0
nanosleep({1, 0}, Process 1480 detached
 <detached ...>
Process 1481 attached
futex(0xd31bc4, FUTEX_WAIT_PRIVATE, 2, NULLProcess 1481 detached
 <detached ...>
Process 1482 attached
accept(8, Process 1482 detached
 <detached ...>
Process 1483 attached
futex(0xd31bc4, FUTEX_WAIT_PRIVATE, 2, NULLProcess 1483 detached
 <detached ...>
Process 1484 attached
accept(9, Process 1484 detached
 <detached ...>
Process 1485 attached
nanosleep({1, 0}, 0x732fedf8)           = 0
nanosleep({1, 0}, 0x732fedf8)           = 0
sysinfo({uptime=349787, loads=[183712, 174848, 188000] totalram=968204288, freeram=157671424, sharedram=151441408, bufferram=1064960} totalswap=0, freeswap=0, procs=238}) = 0
nanosleep({0, 500000000}, Process 1485 detached
 <detached ...>
Process 20463 attached
futex(0xd31bc4, FUTEX_WAIT_PRIVATE, 2, NULLProcess 20463 detached
 <detached ...>
Process 30664 attached
futex(0xd31bc4, FUTEX_WAIT_PRIVATE, 2, NULLProcess 30664 detached
 <detached ...>
Process 30711 attached
futex(0xd31bc4, FUTEX_WAIT_PRIVATE, 2, NULLProcess 30711 detached
 <detached ...>
Process 30786 attached
futex(0xd31bc4, FUTEX_WAIT_PRIVATE, 2, NULLProcess 30786 detached
 <detached ...>
Process 30907 attached
futex(0xd31bc4, FUTEX_WAIT_PRIVATE, 2, NULLProcess 30907 detached
 <detached ...>
Process 30978 attached
futex(0xd31bc4, FUTEX_WAIT_PRIVATE, 2, NULLProcess 30978 detached
 <detached ...>
Process 31078 attached
futex(0xd31bc4, FUTEX_WAIT_PRIVATE, 2, NULLProcess 31078 detached
 <detached ...>
Process 31140 attached
futex(0xd31bc4, FUTEX_WAIT_PRIVATE, 2, NULLProcess 31140 detached
 <detached ...>
Process 31212 attached
futex(0xd31bc4, FUTEX_WAIT_PRIVATE, 2, NULLProcess 31212 detached
 <detached ...>
Process 31334 attached
futex(0xd31bc4, FUTEX_WAIT_PRIVATE, 2, NULLProcess 31334 detached
 <detached ...>
Process 31442 attached
futex(0xd31bc4, FUTEX_WAIT_PRIVATE, 2, NULLProcess 31442 detached
 <detached ...>
Process 31540 attached
futex(0xd31bc4, FUTEX_WAIT_PRIVATE, 2, NULLProcess 31540 detached
 <detached ...>
root@ognpaw:/home/pi#

Interestingly, in the time it took me to figure out the command line to xargs, a few more SBS listeners have spawned.  So maybe the proliferation of SBS listeners is cause by PilotAware-OGN becoming unresponsive.

Code: [Select]

root@ognpaw:/home/pi# ps -T -p 1469
  PID  SPID TTY          TIME CMD
 1469  1469 pts/3    00:00:00 PilotAware-OGN.
 1469  1474 pts/3    00:00:09 PA_httpd
 1469  1479 pts/3    00:00:23 OGN_recv
 1469  1480 pts/3    00:03:39 OGN_send
 1469  1481 pts/3    00:25:09 PA_RxTxRadio
 1469  1482 pts/3    00:00:00 PA_TxSBS1
 1469  1483 pts/3    00:03:53 PA_ParseSBS1
 1469  1484 pts/3    00:00:00 aprs_local
 1469  1485 pts/3    00:07:39 PA_SysMonitor
 1469 20463 pts/3    00:00:26 PA_TxSBS1
 1469 30664 pts/3    00:00:00 PA_httpd_acc
 1469 30711 pts/3    00:00:00 PA_TxSBS1
 1469 30786 pts/3    00:00:00 PA_TxSBS1
 1469 30907 pts/3    00:00:00 PA_TxSBS1
 1469 30978 pts/3    00:00:00 PA_TxSBS1
 1469 31078 pts/3    00:00:00 PA_TxSBS1
 1469 31140 pts/3    00:00:00 PA_TxSBS1
 1469 31212 pts/3    00:00:00 PA_TxSBS1
 1469 31334 pts/3    00:00:00 PA_TxSBS1
 1469 31442 pts/3    00:00:00 PA_TxSBS1
 1469 31540 pts/3    00:00:00 PA_TxSBS1
 1469 31826 pts/3    00:00:00 PA_TxSBS1

And it seems to slowly keep growing:

Code: [Select]

root@ognpaw:/home/pi# ps -T -p 1469 | grep -c TxSBS
15
root@ognpaw:/home/pi# ps -T -p 1469 | grep -c TxSBS
18

I connected back to PilotAware and I did get 1 line printed, so not totally dead:

(14:01:53) FEEDOUT   : SBS Connections = 18

A control X, and it is back up and running aging:

Code: [Select]

^X

@@@ Current time: Sat Jun  1 14:02:39 2019
@@@ Child process is shutting down, a new one will be restarted shortly
@@@ ^R or ^X restarts the child, ^Q quits the server, ^C closes this connection

@@@ @@@ @@@ @@@ @@@
@@@ Received a sigChild for process 1469. The process was killed by signal 15
@@@ Restarting child "./PilotAware-OGN.exe"
@@@ The PID of new child "./PilotAware-OGN.exe" is: 32373
@@@ @@@ @@@ @@@ @@@
./PilotAware-OGN.exe Version 20190524
(14:02:40) MAIN      : Hostid=EB2D4054
(14:02:40) MAIN      : sysdata.ownship.icao=FD4054
(14:02:40) MAIN      : Current Working Directory = /home/pi/rtlsdr-ogn-0.2.6
(14:02:40) MAIN      : APRS_LATITUDE=52.322739
(14:02:40) MAIN      : APRS_LONGITUDE=-0.706224
(14:02:40) MAIN      : APRS_ALTAMSL=301ft
(14:02:40) MAIN      : Item=APRS_ALTGEOID=48
(14:02:40) MAIN      : Item=APRS_SDR_PPM=-0
(14:02:40) MAIN      : Item=APRS_SDR_GAIN=50
(14:02:40) MAIN      : Item=APRS_SDR_FREQCNTR=921.200
(14:02:40) MAIN      : IGNORE RPI_HOSTNAME=ognpaw
(14:02:40) MAIN      : memory:        1024 MB
(14:02:40) MAIN      : processor:     Broadcom BCM2837
(14:02:40) MAIN      : model:         Model B Pi 3
(14:02:40) MAIN      : manufacturer:  Embest
(14:02:40) MAIN      : pcb revision:  2
(14:02:40) MAIN      : revision:      a22082
(14:02:40) MAIN      : Feature Set = 00000003
httpd running on port 80
(14:02:40) INTF-OGN  : Start
(14:02:40) INTF-OGN  : OGN_RECV Connected
(14:02:40) APRS-LOCAL: Listening ...
(14:02:40) TRX-SER   : Configuration Begin
(14:02:40) MONITOR   : SysMonitor start
(14:02:40) INTF-OGN  : APRS_SERVER: APRS.BeaconInterval=10 sec
(14:02:40) INTF-OGN  : APRS_SERVER: Login Paused, Awaiting Station info valid=0 ident=0 ...
TEMPERATURE,0.00,C
(PARSE-FAIL)▒▒▒▒▒▒"APRS.BeaconInterval=10 sec
(PARSE-FAIL)APRS.BeaconInterval=10 sec
(PARSE-FAIL)Settable parameters:
(PARSE-FAIL)Demodulator.DebugPrint=0
(PARSE-FAIL)Demodulator.DetectSNR=10.0 dB
(PARSE-FAIL)Demodulator.ScanMargin=20.0 kHz
(PARSE-FAIL)APRS.BeaconInterval=10 sec
(PARSE-FAIL)APRS.MaxBitError=16 bit(s)
(14:02:40) INTF-OGN  : APRS <- EGBKE>APRS:/140240h5219.36NI00042.37W&/A=000302
(14:02:40) INTF-OGN  : Extract Message 8 chars 0x764a5a20 -> 0x764a5a2d:
(14:02:40) INTF-OGN  : 'EGBKE' -> 'PWEGBKE'
(14:02:41) TRX-SER   : Serial Device = /dev/serial0
(14:02:41) INTF-OGN  : popen->'curl -s http://ipecho.net/plain'
(14:02:41) TRX-SER   : Serial Device = /dev/serial0
(14:02:41) INTF-OGN  : PARSE(1) >APRS:/140240h5219.36NI00042.37W&/A=000302
(14:02:41) INTF-OGN  : Receiver lat=5219.36N(52.32267) lng=42.37W(-0.70617) alt=0
▒▒▒(PARSE-FAIL)APRS.BeaconInterval=300 sec
▒▒▒"(PARSE-FAIL)APRS.BeaconInterval=300 sec
(PARSE-FAIL)Settable parameters:
(PARSE-FAIL)Demodulator.DebugPrint=0
(PARSE-FAIL)Demodulator.DetectSNR=10.0 dB
(PARSE-FAIL)Demodulator.ScanMargin=20.0 kHz
(PARSE-FAIL)APRS.BeaconInterval=300 sec
(PARSE-FAIL)APRS.MaxBitError=16 bit(s)
(14:02:42) MLAT-SRVR : (Connection) WAIT Paused 9
(14:02:43) MLAT-SRVR : (Connection) WAIT Paused 8
(14:02:44) FEEDOUT   : SBS Connections = 1
(14:02:44) MLAT-SRVR : (Connection) WAIT Paused 7
(14:02:45) MLAT-SRVR : (Connection) WAIT Paused 6
(14:02:46) TRX-SER   : Settings Reconfigured Successfully
(14:02:46) BMP       : MPL3115A2 inreset=1
(14:02:46) MLAT-SRVR : (Connection) WAIT Paused 5
(14:02:46) TRX-SER   : Configuration Complete
(14:02:47) BMP       : MPL3115A2 inreset=0
(14:02:47) COMMON    : LOCK: Blocked=0.85469    FEEDOUT(1) ->  radio_OGN
(14:02:47) TRX-RF    : BCN-TX icao=FD4054 acft=1f lat=52.32267 lon=-0.70617 alt=92
(14:02:47) TRX-RF    : PAW-RX icao=4037F8 acft=08 lat=52.49831 lon=-0.20931 alt=1613
(14:02:47) MLAT-SRVR : (Connection) WAIT Paused 4
(14:02:48) MLAT-SRVR : (Connection) WAIT Paused 3
(14:02:49) MLAT-SRVR : (Connection) WAIT Paused 2
(14:02:50) COMMON    : LOCK: Blocked=1.80786  radio_OGN(3) ->    FEEDOUT
(14:02:50) TRX-RF    : PAW-RX icao=4037F8 acft=08 lat=52.49849 lon=-0.20866 alt=1613
(14:02:50) MLAT-SRVR : (Connection) WAIT Paused 1
(14:02:50) TRX-RF    : Error: ------ CRC PASS(2) FAIL(1) ------
(14:02:50) TRX-RF    : expected F4 got FD
24 40 5B DA 0A BC 9F BE 7E 69 51 42 CE 04 46 01 00 55 4E 44 58 00 08 FD
(14:02:51) MLAT-SRVR : (Connection) WAIT Paused 0
(14:02:51) MLAT-SRVR : (Connection) Paused, Awaiting Station Name ...
(14:02:51) MLAT-SRVR : (Connection) Paused, Awaiting Station Position ...
(14:02:51) MLAT-SRVR : PAW Station Name = 'PWEGBKE'
(14:02:51) MLAT-SRVR : WAN Station Addr = 'xxxxxxxxxxx'
(14:02:51) MLAT-SRVR : TCP Attempting MLAT Connection pilotaware.360radar.co.uk:4369 ...
(14:02:51) INTF-OGN  : APRS_SERVER: OGN Station Name = 'EGBKE'
(14:02:51) INTF-OGN  : APRS_SERVER: PAW Station Name = 'PWEGBKE'
(14:02:51) INTF-OGN  : APRS_SERVER: WAN Station Addr = 'xxxxxxxxxxx'
(14:02:51) INTF-OGN  : APRS_SERVER: APRS.BeaconInterval=300 sec
(14:02:51) MLAT-SRVR : TCP MLAT Connected pilotaware.360radar.co.uk:4369
(14:02:51) MLAT-SRVR : (Login) user PWEGBKE pass 15488 vers OGN-R/PilotAware 20190524 filter g/ALL
(14:02:51) MLAT-SRVR : select()=1 tv_sec=9 errno=Success
(14:02:51) MLAT-SRVR : select() rfds=1
(14:02:51) MLAT-SRVR : Read SBS_ENABLE = 0
(14:02:51) MLAT-SRVR : read()=75
(14:02:51) MLAT-SRVR : (Read(75)) # logresp PWEGBKE verified, server 360Radar<CHAR_LF>SBS_ENABLE=0<CHAR_LF>MSG,1,,,,,,,,,,,,<CHAR_LF>
(14:02:51) MLAT-SRVR : (Login) OK
(14:02:51) MLAT-SRVR : (Write) PWEGBKE>APRS:/140240h5219.36NI00042.37W&/A=000302
(14:02:51) MLAT-SRVR : (Write) OK
(14:02:51) MLAT-SRVR : STATUS Connected - Receiving data
(14:02:52) MLAT-SRVR : STATUS 406D7F Latency=(14:02:49.857) latency=2 error=0
DEBUG,MODEC-FILTER-SET,200,MAX,13822064,VDIFF
(14:02:52) TRX-RF    : P3I-TX icao=406D7F acft=1e lat=-0.95030 lon=52.60990 alt=65535 latency=2 (ogn_uplink=0)
(14:02:52) MLAT-SRVR : STATUS 406D7F Latency=(14:02:51.714) latency=0 error=131
(14:02:52) MLAT-SRVR : STATUS 406D7F Time Reverse (Abort Update) 14:02:51 to 14:02:49
(14:02:52) INTF-OGN  : APRS_SERVER: gethostbyname() aprs.glidernet.org
(14:02:52) INTF-OGN  : APRS_SERVER: connect() OK
(14:02:52) TRX-RF    : PAW-RX icao=4037F8 acft=08 lat=52.49920 lon=-0.20603 alt=1612
(14:02:52) MLAT-SRVR : STATUS 405EF5 Latency=(14:02:50.125) latency=2 error=0
(14:02:53) MLAT-SRVR : STATUS 404462 Latency=(14:02:49.840) latency=3 error=0
(14:02:53) MLAT-SRVR : STATUS 4076AC Latency=(14:02:49.509) latency=3 error=0
(14:02:53) MLAT-SRVR : STATUS 405EF5 Latency=(14:02:51.126) latency=2 error=111
(14:02:53) MLAT-SRVR : STATUS 400C30 Latency=(14:02:49.806) latency=3 error=0
(14:02:54) TRX-RF    : P3I-TX icao=404462 acft=1e lat=-0.03560 lon=52.64030 alt=65535 latency=4 (ogn_uplink=0)
(14:02:54) TRX-RF    : P3I-TX icao=4076AC acft=1e lat=-0.06320 lon=52.32560 alt=65535 latency=4 (ogn_uplink=1)
(14:02:54) TRX-RF    : P3I-TX icao=406D7F acft=1e lat=-0.95070 lon=52.61330 alt=65535 latency=2 (ogn_uplink=2)
(14:02:54) TRX-RF    : PAW-RX icao=4037F8 acft=08 lat=52.49957 lon=-0.20469 alt=1613
(14:02:55) MLAT-SRVR : STATUS 4060AE Latency=(14:02:55.723) latency=0 error=0
(14:02:55) TRX-RF    : P3I-TX icao=404462 acft=1e lat=-0.03560 lon=52.64030 alt=65535 latency=5 (ogn_uplink=0)
(14:02:55) TRX-RF    : P3I-TX icao=4076AC acft=1e lat=-0.06320 lon=52.32560 alt=65535 latency=5 (ogn_uplink=1)
(14:02:55) TRX-RF    : P3I-TX icao=406D7F acft=1e lat=-0.95070 lon=52.61330 alt=65535 latency=3 (ogn_uplink=2)
(14:02:56) MLAT-SRVR : STATUS 405EF5 Latency=(14:02:52.632) latency=3 error=98
(14:02:56) MLAT-SRVR : STATUS 401652 Latency=(14:02:54.276) latency=2 error=0
(14:02:56) TRX-RF    : PAW-RX icao=4037F8 acft=08 lat=52.49994 lon=-0.20336 alt=1613
(14:02:57) MLAT-SRVR : STATUS 400C30 Latency=(14:02:55.503) latency=1 error=531
DEBUG,MODEC-FILTER-HLD,200,MAX
(14:02:57) MLAT-SRVR : STATUS 40389B Latency=(14:02:55.588) latency=1 error=0
(14:02:57) MLAT-SRVR : STATUS 404462 Latency=(14:02:53.643) latency=3 error=148
(14:02:57) MLAT-SRVR : STATUS 406C0D Latency=(14:02:55.736) latency=1 error=0
(14:02:57) MLAT-SRVR : STATUS 4076AC Latency=(14:02:55.125) latency=2 error=195
(14:02:57) INTF-OGN  : APRS_SERVER: (send-login) user PWEGBKE pass 25017 vers OGN-R/PilotAware 20190524 filter g/ALL
(14:02:57) INTF-OGN  : APRS_SERVER: (send-login) user PWEGBKE pass 25017 vers OGN-R/PilotAware 20190524 filter g/ALL
 - SENT
(14:02:57) INTF-OGN  : APRS_SERVER: aprs_ident = >APRS:/140240h5219.36NI00042.37W&/A=000302
(14:02:57) INTF-OGN  : APRS_SERVER: (send-ident1) PWEGBKE>APRS:/140240h5219.36NI00042.37W&/A=000302
(14:02:57) INTF-OGN  : APRS_SERVER: (send-ident2) PWEGBKE>APRS:>140257h v20190524 OGN-R/PilotAware
(14:02:57) INTF-OGN  : APRS_SERVER: (recv) # aprsc 2.1.4-g408ed49
TEMPERATURE,69.30,C
(14:02:57) INTF-OGN  : APRS_SERVER: (recv) # logresp PWEGBKE verified, server GLIDERN2
(14:02:57) TRX-RF    : P3I-TX icao=404462 acft=1e lat=-0.03500 lon=52.64230 alt=65535 latency=3 (ogn_uplink=0)
(14:02:57) TRX-RF    : P3I-TX icao=4076AC acft=1e lat=-0.06530 lon=52.32460 alt=65535 latency=2 (ogn_uplink=1)
(14:02:57) TRX-RF    : P3I-TX icao=406D7F acft=1e lat=-0.95070 lon=52.61330 alt=65535 latency=5 (ogn_uplink=2)
(14:02:58) MLAT-SRVR : STATUS 404D02 Latency=(14:02:55.946) latency=2 error=0
(14:02:58) MLAT-SRVR : STATUS 40657A Latency=(14:02:55.047) latency=3 error=0
(14:02:58) TRX-RF    : PAW-RX icao=4037F8 acft=08 lat=52.50030 lon=-0.20204 alt=1613
(PARSE-FAIL)HTTP_Server.Exec() ... Client from 63.143.42.251:53797
(14:02:58) MLAT-SRVR : STATUS 406922 Latency=(14:02:53.823) latency=4 error=0
(14:02:58) MLAT-SRVR : STATUS 4021C2 Latency=(14:02:55.086) latency=3 error=0
(14:02:58) MLAT-SRVR : STATUS 401652 Latency=(14:02:57.710) latency=0 error=315
(14:02:59) TRX-RF    : P3I-TX icao=404462 acft=1e lat=-0.03500 lon=52.64230 alt=65535 latency=5 (ogn_uplink=0)
(14:02:59) TRX-RF    : P3I-TX icao=4076AC acft=1e lat=-0.06530 lon=52.32460 alt=65535 latency=4 (ogn_uplink=1)
(14:02:59) TRX-RF    : P3I-TX icao=406922 acft=1e lat=-0.04250 lon=52.69170 alt=65535 latency=5 (ogn_uplink=2)
(14:02:59) TRX-RF    : P3I-TX icao=4021C2 acft=1e lat=-0.52500 lon=52.64980 alt=65535 latency=4 (ogn_uplink=3)
(14:02:59) TRX-RF    : P3I-TX icao=40657A acft=1e lat=-0.15520 lon=52.21520 alt=65535 latency=4 (ogn_uplink=4)
(14:02:59) TRX-RF    : P3I-TX icao=406D7F acft=1e lat=-0.95070 lon=52.61330 alt=65535 latency=7 (ogn_uplink=5)
(14:03:00) TRX-RF    : PAW-RX icao=4037F8 acft=08 lat=52.50049 lon=-0.20137 alt=1612
(14:03:00) MLAT-SRVR : STATUS 405EF5 Latency=(14:02:59.501) latency=0 error=317
(14:03:00) MLAT-SRVR : STATUS 405EF5 Latency=(14:02:59.706) latency=0 error=144
(14:03:00) MLAT-SRVR : STATUS 400C30 Latency=(14:02:58.009) latency=2 error=750
(14:03:00) MLAT-SRVR : STATUS 406922 Latency=(14:02:58.306) latency=2 error=355
(14:03:00) MLAT-SRVR : STATUS 4076AC Latency=(14:02:59.188) latency=1 error=146
(14:03:00) MLAT-SRVR : STATUS 406D7F Latency=(14:02:59.182) latency=1 error=364
(14:03:01) TRX-RF    : P3I-TX icao=404462 acft=1e lat=-0.03500 lon=52.64230 alt=65535 latency=7 (ogn_uplink=0)
(14:03:01) TRX-RF    : P3I-TX icao=4076AC acft=1e lat=-0.06630 lon=52.32380 alt=65535 latency=2 (ogn_uplink=1)
(14:03:01) TRX-RF    : P3I-TX icao=406922 acft=1e lat=-0.04690 lon=52.68900 alt=65535 latency=3 (ogn_uplink=2)


Cheers
Kev

[Pidea]

If you've got this on the internet and you've got ports 80 / 8080 and 8081 open then it's highly likely that your traffic is being siphoned off.  It's really easy to build a Google search query to find VRS instances online and then pull their data without the owners knowing.

VRS is great when running on a closed network with a small handful of users but it's very inefficient when run on the internet.  For every single connection to VRS it spawns a new thread.  Even on a decent server it'll start to slow down when it gets to between 50 and 100 simultaneous users.  That's why 360Radar runs through load balancers to spread the load across multiple servers and why I've also removed a lot of the duties from VRS such as the image generation as that is extraordinarily inefficient.

If you must run it online then password protect your feed to stop people leeching your data.

[Kevin W]

Hi Pidea, port 9000 that the VRS server runs on is not exposed to the internet, just the 3 status ports.

Thanks
Kev

[Pidea]

So what's on 8080, 8081 and 80 then ?

[Kevin W]

Hi

As per a standard OGN-R station, 8080 is the OGN-RF status page, 8081 is the OGN-Decode status page and 80 (port mapped to 8082) is the PilotAware-OGN status page.

Cheers
Kev