DTTV pid stats logging

On 04 Oct, wrote:
In article , John
Legon wrote:


As an experiment, last night I made two 'long' traffic recordings (BBCA
Scotland). One 30 mins from Durris, one 90 mins from Angus.

I'll examine these and see if there are any signs of problems. But not
were immediately evident.

A preliminary examination of the Angus BBCA (Scottish) data I got shows
results similar to those I've been getting previously from both versions of
dvbtraffic.

The audio rates and the total rate (all PID values summed) seem to hold
steady with a measured level of variation around 0.1% or less. This is on
timescales for counts/averages from 5 sec to 5 min for a duration of 90
mins.

So, no sign of a problem so far. However I'll continue to poke around and
investigate this.

Slainte,

Jim

--
Please use the address on the audiomisc page if you wish to email me.
Electronics http://www.st-and.ac.uk/~www_pa/Scot...o/electron.htm
Armstrong Audio http://www.audiomisc.co.uk/Armstrong/armstrong.html
Audio Misc http://www.audiomisc.co.uk/index.html

In article , John Legon
wrote:
Jim Lesurf wrote:
I've been making a number of test recordings using both the 'standard'
dvbtraffic and my hacked version for some days. I've not yet
encountered any sign of the data stream from my 290e either not
starting at a packet head, or having a jump in synch during the stream
due to a packet having other than 188 bytes.

So I've suspected that this is either set by the hardware doing the
tuning and streaming, or is ensured by having a signal strong enough
for reception.

I would be inclined to create a test version of the program which tried
to get out of sync by reading some arbitrary fraction of a packet at a
time instead the full 188 bytes. My guess is that the successive reads
will still start on packet boundaries so you may not see any difference
in the results.

You would have to explain the reasoning behind your guess. (Ideally, do
what you suggest yourself and report the results along with your
reasoning.) The difficulty with 'hidden' mechanisms is that any test needs
to be able to address the details of how/why it is hidden to hope to
succeed.

I've proceeded thus far on the basis of looking to see if I can find any
sign of a problem. So far, the stats show no clear sign. e.g. the audio
rates remain steady. However this is with recordings made with the RX
having a good UHF signal. I may find otherwise when I try recording traffic
with a deliberately degraded UHF signal. And there may be *brief* periods
of lost synch that correct and have little effect over a period of, say,
seconds or longer. I'm not sure what this would do to the FEC and other
correction schemes. But we need to bear in mind that any DTTV RX has to do
a fair bit of work to convert the incoming COFDM into a packetted output
stream.

Similarly, there may be a problem when the signal is lousy. But when the
input is good, our old friend "Digital Cliff" may mean problems are so rare
as to be of no more than ahem academic interest.

If the 290e or interface are 'covering up' problems, they may well be doing
so by simply putting junk into a series of 188-byte packets. The result is,
externally, a correctly packeted stream with a header every 188 bytes and a
payload, etc, inside. The problem then is not synch, but packet contents. -
i.e. that some payloads may be junk, or an error report.

If so, the way to tell is to see if the result decodes as useable data when
played, etc. Not clear to me at present why what you propose will deal with
this.

Slainte,

Jim

--
Please use the address on the audiomisc page if you wish to email me.
Electronics http://www.st-and.ac.uk/~www_pa/Scot...o/electron.htm
Armstrong Audio http://www.audiomisc.co.uk/Armstrong/armstrong.html
Audio Misc http://www.audiomisc.co.uk/index.html

On 04 Oct, wrote:
On 04 Oct, wrote:
In article , John
Legon wrote:


As an experiment, last night I made two 'long' traffic recordings
(BBCA Scotland). One 30 mins from Durris, one 90 mins from Angus.

I'll examine these and see if there are any signs of problems. But not
were immediately evident.

A preliminary examination of the Angus BBCA (Scottish) data I got shows
results similar to those I've been getting previously from both versions
of dvbtraffic.

To help people assess this I've put some rough results up at

http://jcgl.orpheusweb.co.uk/temp/bbca_stats.html

This is the 90 min Angus traffic data, averaged into 5-min duration bins
from the 5 second bins recorded.

The main significance here is the level of steadyness in the audio (and
total pid-count) rate values. I doubt my analysis program is totally
debugged yet, but the results look promising.

There seems no sign of any persistent desynch problems so far as I can see.
Although their may be brief problems. The level of variation seems to me to
be what I'd expect from the timing 'jitter' of my computer clock against
the clock defined by the mux symbol rate. (Using the 'run for 128 packets
between timechecks' and 'do counts over circa 5 sec blocks' as currently
set in the ROX app version I've done of dbvtraffic.)

If puzzled by only seeing one line per audio graph... the solution is that
all the rates either have one value or t'other. They track very well. This
is pretty much what we should expect AIUI. But it means each line 'hides' a
number of others that are drawn exactly under them! :-)

Slainte,

Jim

--
Please use the address on the audiomisc page if you wish to email me.
Electronics http://www.st-and.ac.uk/~www_pa/Scot...o/electron.htm
Armstrong Audio http://www.audiomisc.co.uk/Armstrong/armstrong.html
Audio Misc http://www.audiomisc.co.uk/index.html

Jim Lesurf wrote:
In article , John Legon
wrote:
Jim Lesurf wrote:
I've been making a number of test recordings using both the 'standard'
dvbtraffic and my hacked version for some days. I've not yet
encountered any sign of the data stream from my 290e either not
starting at a packet head, or having a jump in synch during the stream
due to a packet having other than 188 bytes.

So I've suspected that this is either set by the hardware doing the
tuning and streaming, or is ensured by having a signal strong enough
for reception.

I would be inclined to create a test version of the program which tried
to get out of sync by reading some arbitrary fraction of a packet at a
time instead the full 188 bytes. My guess is that the successive reads
will still start on packet boundaries so you may not see any difference
in the results.

You would have to explain the reasoning behind your guess. (Ideally, do
what you suggest yourself and report the results along with your
reasoning.)

Agreed, this is something I should probably try myself to satisfy my own
curiosity, but I don't have a working Linux/dvb system. However, since
it is evident that dvbtraffic cannot of itself bring itself into sync
with the TS stream, it seems to follow that the Linux driver for dvr
devices is delivering TS packets which always begin with a sync byte, so
dvbtraffic never sees anything different.

The difficulty with 'hidden' mechanisms is that any test needs
to be able to address the details of how/why it is hidden to hope to
succeed.

It's not a question of addressing details, but of establishing the
behaviour in principle. I wouldn't hesitate to change BSIZE from 188 to
say 94 bytes, so that half a packet would be requested with each 'read'.
I would expect the remaining half of each packet to be discarded, so
that the next 'read' would still begin with the next sync byte. Since
the PID comes directly after the sync byte, your program output would
not be affected.

[...]
If the 290e or interface are 'covering up' problems, they may well be doing
so by simply putting junk into a series of 188-byte packets. The result is,
externally, a correctly packeted stream with a header every 188 bytes and a
payload, etc, inside. The problem then is not synch, but packet contents. -
i.e. that some payloads may be junk, or an error report.

If so, the way to tell is to see if the result decodes as useable data when
played, etc. Not clear to me at present why what you propose will deal with
this.

My proposal was intended simply to provide a confirmation of the way the
stream is handled. Or not, as the case may be. :-)

Johann Klammer wrote:

Tried something like this, too. Just uploaded it.
Can be found he
https://github.com/klammerj/dvb-vult...aster/tsverify
(should be one line..)

Looks interesting. I also have some code that dumps some data about ts
flags and some other things to help cutting. It also does some
(crude/lazy) cont checking so it's nice to see something else to compare
results.

I've never seen a desync so far in any recording I've looked at and some
of them have packets with the Transport Error flag set.

I don't know if it's just my old bash version, but "$*" in the runme.sh
means I have to omit the space in the -i path/to/infilename (which means
tab doesn't autocomplete).

"[email protected]" or either way without quotes works for me.

On Thu, 04 Oct 2012 09:42:17 +0100, Jim Lesurf wrote:

But I admit I sometimes leave in commented
out code that was used during test or experiment in case I want it
again. However in this instance "not guilty, m'lud!" :-)

I *always* leave debugging code in production programs, but under the
control of a debug variable I control with a command line option. It
frequently helps to triage user finger trouble from actual bugs and
anyway the impact of:

if (debug)
{
/* debugging code */
}

on run-time performance is vanishingly small with any modern CPU that
implements predictive branching. It was negligible on older machines too.


--
[email protected] | Martin Gregorie
gregorie. | Essex, UK
org |

Andy Furniss wrote:
Johann Klammer wrote:

Tried something like this, too. Just uploaded it.
Can be found he
https://github.com/klammerj/dvb-vult...aster/tsverify
(should be one line..)

Looks interesting. I also have some code that dumps some data about ts
flags and some other things to help cutting. It also does some
(crude/lazy) cont checking so it's nice to see something else to compare
results.

I've never seen a desync so far in any recording I've looked at and some
of them have packets with the Transport Error flag set.
I've never seen a desync either. It was used for testing homegrown
streaming program, so it tries to be over-defensive.


I don't know if it's just my old bash version, but "$*" in the runme.sh
means I have to omit the space in the -i path/to/infilename (which means
tab doesn't autocomplete).

"[email protected]" or either way without quotes works for me.
Thanks. Changed it.

Jim Lesurf wrote:
I've proceeded thus far on the basis of looking to see if I can find any
sign of a problem. So far, the stats show no clear sign. e.g. the audio
rates remain steady. However this is with recordings made with the RX
having a good UHF signal. I may find otherwise when I try recording traffic
with a deliberately degraded UHF signal. And there may be *brief* periods
of lost synch that correct and have little effect over a period of, say,
seconds or longer. I'm not sure what this would do to the FEC and other
correction schemes. But we need to bear in mind that any DTTV RX has to do
a fair bit of work to convert the incoming COFDM into a packetted output
stream.

Similarly, there may be a problem when the signal is lousy. But when the
input is good, our old friend "Digital Cliff" may mean problems are so rare
as to be of no more than ahem academic interest.

If the 290e or interface are 'covering up' problems, they may well be doing
so by simply putting junk into a series of 188-byte packets. The result is,
externally, a correctly packeted stream with a header every 188 bytes and a
payload, etc, inside. The problem then is not synch, but packet contents. -
i.e. that some payloads may be junk, or an error report.

If so, the way to tell is to see if the result decodes as useable data when
played, etc.

One possible way to tell is by looking at the transport_error_indicator
flag which is present in each TS packet. My satellite box never sets
this flag so it isn't much use for that, but I discovered last night
that my DTT tuner stick does set the flag, so packets with uncorrected
errors can be identified when I record a mux on my PC (running XP).

For the "BBC" mux and an average signal this morning, my analyser code
found 5 packets with errors in the first 10 packets, but no errors at
all in the next 200,000 packets.

For the next 200,000 packets, during the recording of which I briefly
disconnected the aerial, there were 1706 packets with at least one
uncorrected error.

The "clean" section of the recording contained 76 unique PIDs, but when
the faulty section was included in the count the number of apparent PIDs
rose to 247.

It follows from this that one really needs to exclude packets in which
the TEI flag is set when collecting PID data, while an exact count of
the number of packets with errors can be obtained.

In article , John
Legon
wrote:
[snip]

One possible way to tell is by looking at the transport_error_indicator
flag which is present in each TS packet.

Agreed. However I've assumed that when this happens the output from tzap
will show a non-zero count for 'uncorrected errors'. Can you say from the
observations you report? It is one reason I'd like to be able to 'tee' the
output from tzap to a log file as well as see it onscreen whilst recording.

My satellite box never sets this flag so it isn't much use for that, but
I discovered last night that my DTT tuner stick does set the flag, so
packets with uncorrected errors can be identified when I record a mux on
my PC (running XP).

Which tuner?

For the "BBC" mux and an average signal this morning, my analyser code
found 5 packets with errors in the first 10 packets, but no errors at
all in the next 200,000 packets.

I have the impression that the tuner takes a short time to get its act
together after being given a tzap. At present I have a 2 second delay
between tzap and starting traffic logging to try and allow for this.
However I've noticed a few times a tiny change in PID stats for the first
part of a run. So maybe this needs to be longer.

For the next 200,000 packets, during the recording of which I briefly
disconnected the aerial, there were 1706 packets with at least one
uncorrected error.

I'd be interested to know what signal level, etc, you had at the time.

Here, for Angus on my main antenna I get '0xffff' signal level all the time
from BBCA, and can even shove more than 10dB of pad in the way and still
get this. I also get essentially continuous error levels of 0. However I
get poorer results from Durris on the small antenna, and plan to do some
tests on the effect of reducing the signal level.

The "clean" section of the recording contained 76 unique PIDs, but when
the faulty section was included in the count the number of apparent PIDs
rose to 247.

247 / 200,000 is curiously close to my estimates that the rates which
should be 'constant' flicker a bit at about the 0.1% level.

What I do see during logging is occasional sections where there is a burst
of many 'rare' PID values. At present I have no idea what causes these, and
if it they are real metadata, or the symptom of an error like a brief
desynch.

It follows from this that one really needs to exclude packets in which
the TEI flag is set when collecting PID data, while an exact count of
the number of packets with errors can be obtained.

For ideal precision, yes. However if I'm getting rate values that are
better than say 1% accurate I find that OK for my purposes.

TBH I'm more concerned about the distinction between the 'outer' (packet)
bitrate and the 'inner' (payload) rates. In addition to some packets being
errors, there are also questions like the changes in header occupation by
the metadata. To look at that properly probably does required recording the
entire mux for complex analysis. However I am not sure if even Ray goes
this far. (Indeed, he may not be sure as he uses closed-source software.)

At some point I'll put up an updated version of my ROX app (and also one or
two others). They anyone who wants can improve or adapt (or laugh at ;- )
them as they wish.

Slainte,

Jim

--
Please use the address on the audiomisc page if you wish to email me.
Electronics http://www.st-and.ac.uk/~www_pa/Scot...o/electron.htm
Armstrong Audio http://www.audiomisc.co.uk/Armstrong/armstrong.html
Audio Misc http://www.audiomisc.co.uk/index.html

Jim Lesurf wrote:

I'd like to be able to 'tee' the
output from tzap to a log file as well as see it onscreen whilst recording.

Any reason you can't do exactly that?