DTTV pid stats logging

In article ,
Jim Lesurf wrote:

I'll see what the consensus is on the duff 'contunue;'. If no-one has a
good word for it, I'll remove if from the version I've written.

Many compilers will warn about unreachable code like this if you
compile with suitable flags (e.g. -Wunreachable-code for gcc). Of
course, this may result in spurious warnings but it's worth doing from
time to time and on code you are importing for the first time - I have
found bugs in code that had been "working" for years just by compiling
with a more loquacious compiler.

-- Richard

Jim Lesurf wrote:
I'll see what the consensus is on the duff 'contunue;'. If no-one has a
good word for it, I'll remove if from the version I've written.

It's hard to imagine how anyone could have a good word to say about the
'duff continue'. Simply taking it out won't solve the problem, however,
since the next block of code will take the file/stream pointer past the
sync byte, which therefore won't be at the start of the next 188 bytes
that are read into the buffer.

Furthermore, there's no guarantee that the next 0x47 in the stream is a
sync byte, since the same value byte can occur in the payload. So you
would need to locate say, five sync bytes in a row at the correct 188
byte intervals to be certain that you are looking at real TS packets and
not meaningless blocks of data. The real (as opposed to imaginary) PIDs
can then be extracted.

In article , John
Legon
wrote:
Jim Lesurf wrote:
I'll see what the consensus is on the duff 'contunue;'. If no-one has
a good word for it, I'll remove if from the version I've written.

It's hard to imagine how anyone could have a good word to say about the
'duff continue'. Simply taking it out won't solve the problem, however,
since the next block of code will take the file/stream pointer past the
sync byte, which therefore won't be at the start of the next 188 bytes
that are read into the buffer.

So why does the code seem to work OK now? So far as I can tell thus far it
gives the same results as a pidscan of a recorded mux. I'm curious since
the continue does seem to mean the program - so far as I know - *never*
does what I'd be removing. But I may well not be understanding.

I'm quite happy to leave the disputed code in place. So far as I can tell,
all that means in practice is a tiny bit of memory occupied uselessly.

Furthermore, there's no guarantee that the next 0x47 in the stream is a
sync byte, since the same value byte can occur in the payload. So you
would need to locate say, five sync bytes in a row at the correct 188
byte intervals to be certain that you are looking at real TS packets and
not meaningless blocks of data. The real (as opposed to imaginary)
PIDs can then be extracted.

When I've scanned recordings I haven't yet encountered any occasions where
the data isn't in 188-byte packets. So I'm wondering: Are you saying that
reception with some uncorrected errors will insert sections that are not an
integer mulitple of 188 bytes between legitimate packets?

Slainte,

Jim

--
Please use the address on the audiomisc page if you wish to email me.
Electronics http://www.st-and.ac.uk/~www_pa/Scot...o/electron.htm
Armstrong Audio http://www.audiomisc.co.uk/Armstrong/armstrong.html
Audio Misc http://www.audiomisc.co.uk/index.html

Jim Lesurf wrote:
In article , John
Legon
wrote:
Jim Lesurf wrote:
I'll see what the consensus is on the duff 'contunue;'. If no-one has
a good word for it, I'll remove if from the version I've written.

It's hard to imagine how anyone could have a good word to say about the
'duff continue'. Simply taking it out won't solve the problem, however,
since the next block of code will take the file/stream pointer past the
sync byte, which therefore won't be at the start of the next 188 bytes
that are read into the buffer.

So why does the code seem to work OK now? So far as I can tell thus far it
gives the same results as a pidscan of a recorded mux. I'm curious since
the continue does seem to mean the program - so far as I know - *never*
does what I'd be removing. But I may well not be understanding.

As far as I can see, the code will work correctly as written provided
that the first byte received when the stream is opened is a sync byte.
The code that is supposed to handle the contrary situation is then never
reached. If there is some mechanism that guarantees that the first byte
is a sync byte then the program should work OK.

My own recordings of transport streams do begin with the sync byte, so
the program would work with these.

I'm quite happy to leave the disputed code in place. So far as I can tell,
all that means in practice is a tiny bit of memory occupied uselessly.

I wouldn't be happy about this because the presence of junk code casts
doubt on the integrity of the program as a whole. But if it works for
you...

Furthermore, there's no guarantee that the next 0x47 in the stream is a
sync byte, since the same value byte can occur in the payload. So you
would need to locate say, five sync bytes in a row at the correct 188
byte intervals to be certain that you are looking at real TS packets and
not meaningless blocks of data. The real (as opposed to imaginary)
PIDs can then be extracted.

When I've scanned recordings I haven't yet encountered any occasions where
the data isn't in 188-byte packets.

I haven't either, but have only tested a couple of recordings thus far.

So I'm wondering: Are you saying that
reception with some uncorrected errors will insert sections that are not an
integer mulitple of 188 bytes between legitimate packets?

I'm not saying that. What I am saying is that if the program starts off
out of sync, or maybe becomes out of sync because of uncorrected errors,
it doesn't provide a logical means of getting back into sync by
locating the start of a TS packet. It reads 188 bytes into a buffer,
checks to see whether the first byte is the sync byte, and if not, it
reads the next 188 bytes from the stream and looks again. But if the
first read wasn't aligned with a TS packet, then successive reads won't
be either! However, there's a chance of it hitting a byte 0x47 in the
payload, and in this case the code will assume that it's found the sync
byte and will construct a PID erroneously from the two following bytes.

John Legon wrote:

So I'm wondering: Are you saying that
reception with some uncorrected errors will insert sections that are
not an
integer mulitple of 188 bytes between legitimate packets?

I'm not saying that. What I am saying is that if the program starts off
out of sync, or maybe becomes out of sync because of uncorrected errors,
it doesn't provide a logical means of getting back into sync by
locating the start of a TS packet. It reads 188 bytes into a buffer,
checks to see whether the first byte is the sync byte, and if not, it
reads the next 188 bytes from the stream and looks again. But if the
first read wasn't aligned with a TS packet, then successive reads won't
be either! However, there's a chance of it hitting a byte 0x47 in the
payload, and in this case the code will assume that it's found the sync
byte and will construct a PID erroneously from the two following bytes.

When I first saw the continue in dvbtraffic I didn't really see it as an
issue. A bit untidy, maybe, and the failing sync code should have been
removed rather than bypassed, but it just looked like some left over
experiment.

AIUI the v4l layer will deliver 188 byte packets - but they may be
corrupt, so skipping to the next packet if the sync byte is trashed is
logical given that info.

In ,
John Legon wrote:

As far as I can see, the code will work correctly as written provided
that the first byte received when the stream is opened is a sync byte.
The code that is supposed to handle the contrary situation is then never
reached. If there is some mechanism that guarantees that the first byte
is a sync byte then the program should work OK.

IME with reading data directly from DVB devices in Linux it does always
start on a packet boundary, but I don't know how it handles corruption.

--
TH * http://www.realh.co.uk

Andy Furniss wrote:
John Legon wrote:

So I'm wondering: Are you saying that
reception with some uncorrected errors will insert sections that are
not an
integer mulitple of 188 bytes between legitimate packets?

I'm not saying that. What I am saying is that if the program starts off
out of sync, or maybe becomes out of sync because of uncorrected errors,
it doesn't provide a logical means of getting back into sync by
locating the start of a TS packet. It reads 188 bytes into a buffer,
checks to see whether the first byte is the sync byte, and if not, it
reads the next 188 bytes from the stream and looks again. But if the
first read wasn't aligned with a TS packet, then successive reads won't
be either! However, there's a chance of it hitting a byte 0x47 in the
payload, and in this case the code will assume that it's found the sync
byte and will construct a PID erroneously from the two following bytes.

When I first saw the continue in dvbtraffic I didn't really see it as an
issue. A bit untidy, maybe, and the failing sync code should have been
removed rather than bypassed, but it just looked like some left over
experiment.

AIUI the v4l layer will deliver 188 byte packets - but they may be
corrupt, so skipping to the next packet if the sync byte is trashed is
logical given that info.



Tried something like this, too. Just uploaded it.
Can be found he
https://github.com/klammerj/dvb-vult...aster/tsverify
(should be one line..)

In article , John
Legon
wrote:


So why does the code seem to work OK now? So far as I can tell thus
far it gives the same results as a pidscan of a recorded mux.

As far as I can see, the code will work correctly as written provided
that the first byte received when the stream is opened is a sync byte.

Agreed.

The code that is supposed to handle the contrary situation is then never
reached. If there is some mechanism that guarantees that the first
byte is a sync byte then the program should work OK.

Yes, again, agreed.

My own recordings of transport streams do begin with the sync byte, so
the program would work with these.


I'm quite happy to leave the disputed code in place. So far as I can
tell, all that means in practice is a tiny bit of memory occupied
uselessly.

I wouldn't be happy about this because the presence of junk code casts
doubt on the integrity of the program as a whole. But if it works for
you...

So far as I'm concerned, programs come in two forms. Those with known bugs
and oddities, and those that only have undetected bugs and oddities. :-)


When I've scanned recordings I haven't yet encountered any occasions
where the data isn't in 188-byte packets.

I haven't either, but have only tested a couple of recordings thus far.

I've been making a number of test recordings using both the 'standard'
dvbtraffic and my hacked version for some days. I've not yet encountered
any sign of the data stream from my 290e either not starting at a packet
head, or having a jump in synch during the stream due to a packet having
other than 188 bytes.

So I've suspected that this is either set by the hardware doing the tuning
and streaming, or is ensured by having a signal strong enough for
reception.

Given this as an assumption, I'd guess that the 'contunue;' was included
when the writer decided checking wasn't actually needed in practice.

What I don't know, is how reliable this assumption may be, and how it may
be affected by what circumstances.

As an experiment, last night I made two 'long' traffic recordings (BBCA
Scotland). One 30 mins from Durris, one 90 mins from Angus.

I'll examine these and see if there are any signs of problems. But not were
immediately evident.

If none surface, I'll experiment with more traffic logging, but
deliberately upset reception by using a variable attenuator to wind down
the input UHF level to get a significant level of uncorrected errors or
loss of FEC LOCK. This will let me see if the hardware still gives an
(externally) correct stream of 188 byte packets even when they may contain
garbage from the POV of getting a useable video/audio output.

From my POV what seems like a good overally check is as follows:

If the stream at some point has a chunk that offsets the packet headers by
a value other than 188 bytes, then it will desynch the output. In effect
this will 'randomise' the stats for how often each (apparent) PID values
appear. Since the audio rates are essentially constant, and the total is
essentially constant. These would start hopping about and probably move to
medium-term averages approximate to being 1/number-of-PIDs of the total
rate. So such an 'enduring' loss of synch would be fairly obvious.

By eye and some analysis I've not seen anything like that so far. But the
longer recordings will give me more data to test.

Of course, a loss of synch might be temporary, and correct after a small
number of packets. That would be harder to detect, but also less serious in
its effects. To test that I could just add back in an active 'print a
warning if the header flag isn't correct'.

My suspicion is that short loss of synch may be unlikely if the tuner is
working OK. This is because I assume the synch timing is set by the symbol
rate clock of the mux modulation. So whilst the tuner is locked, and has,
say a PLL, to roll over a reasonable time, any short term breaks in data
won't lose the synch. Just sometimes mean some junk inside some packets.
However this is just my reasoning at present, so I'll have to test to see
what actually happens.

BTW a question: Does DAB use the same 188-byte packet stream approach?
And does that mean much the same method can be used to make DAB
recordings and traffic analysis?

Slainte,

Jim

--
Please use the address on the audiomisc page if you wish to email me.
Electronics http://www.st-and.ac.uk/~www_pa/Scot...o/electron.htm
Armstrong Audio http://www.audiomisc.co.uk/Armstrong/armstrong.html
Audio Misc http://www.audiomisc.co.uk/index.html

In article ,
Andy Furniss wrote:

When I first saw the continue in dvbtraffic I didn't really see it as an
issue. A bit untidy, maybe, and the failing sync code should have been
removed rather than bypassed, but it just looked like some left over
experiment.

That was my reaction when John raised this. It looks odd, but maybe just a
'commented out' bit of test code that was found not to be needed. The
though may have been to leave it in place in case it was wanted again when
a problem showed up. If so, commenting out seems better as it would avoid
pointless compilation. But I admit I sometimes leave in commented out code
that was used during test or experiment in case I want it again. However in
this instance "not guilty, m'lud!" :-)

Slainte,

Jim

--
Please use the address on the audiomisc page if you wish to email me.
Electronics http://www.st-and.ac.uk/~www_pa/Scot...o/electron.htm
Armstrong Audio http://www.audiomisc.co.uk/Armstrong/armstrong.html
Audio Misc http://www.audiomisc.co.uk/index.html

Jim Lesurf wrote:
I've been making a number of test recordings using both the 'standard'
dvbtraffic and my hacked version for some days. I've not yet encountered
any sign of the data stream from my 290e either not starting at a packet
head, or having a jump in synch during the stream due to a packet having
other than 188 bytes.

So I've suspected that this is either set by the hardware doing the tuning
and streaming, or is ensured by having a signal strong enough for
reception.

I would be inclined to create a test version of the program which tried
to get out of sync by reading some arbitrary fraction of a packet at a
time instead the full 188 bytes. My guess is that the successive reads
will still start on packet boundaries so you may not see any difference
in the results.