Frequently Asked Questions (FAQ) about the "LIVE555 Streaming Media" libraries

What is the typical control flow within an application that uses these libraries, and what is the role of the various "Source" and "Sink" classes in the "liveMedia" library?

Applications are event-driven, using an event loop "TaskScheduler::doEventLoop()" that works basically as follows:

    while (1) {
        find a task that needs to be done (by looking on the delay queue,
                and the list of network read handlers);
        perform this task;
    }

Also, before entering this loop, applications will typically call

    someSinkObject->startPlaying();

for each sink, to start generating tasks that need to be done.

Data passes through a chain of 'source's and 'sink's - e.g.,

    'source1' -> 'source2' (a filter) -> 'source3' (a filter) -> 'sink'

(Sources that receive data from other sources are also called "filters".)

Whenever a module (a 'sink' or one of the intermediate filter 'source's) wants to get more data, it calls "FramedSource::getNextFrame()" on the module that's to its immediate left. This is implemented by the pure virtual function "FramedSource::doGetNextFrame()", that is implemented by each 'source' module.

Each 'source' module's implementation of "doGetNextFrame()" works by arranging for an 'after getting' function to be called (from an event handler) when new data becomes available for the caller.

Note that the flow of data from 'sources' to 'sinks' happens within each application, and doesn't necessarily correspond to the sending or receiving of network packets. For example, a server application (such as "testMP3Streamer") that sends RTP packets will do so using one or more "RTPSink" modules. These "RTPSink" modules receive data from other, "*Source" modules (e.g., to read data from a file), and, as a side effect, transmit RTP packets.

How can I use this code within a specialized environment (such as an embedded system, or a GUI toolkit)?

People usually do this by developing their own subclasses of the "UsageEnvironment" and "TaskScheduler" abstract base classes (see "UsageEnvironment/include/UsageEnvironment.hh"). Note that the released source code includes one particular implementation of these classes: the "BasicUsageEnvironment" library. This uses the Unix (or Windows) console for I/O, and so allows you to develop applications that you can run in a conventional console environment - e.g., for prototyping and debugging. Then, by using your own custom subclasses of "UsageEnvironment" and "TaskScheduler" (i.e., instead of "BasicUsageEnvironment" and "BasicTaskScheduler"), the same code will run, unchanged, in your custom environment.

In particular, to use the code within a GUI toolkit, your "TaskScheduler" subclass's implementation of "doEventLoop()" should be integrated with the GUI toolkit's own event handling mechanism.

Is this code 'thread safe'? I.e., can it be accessed by more than one thread at the same time?

Short answer: No. As noted above, the code assumes a single thread of execution, using an event loop - rather than multiple threads - for concurrency. This generally makes the code much easier to debug, and much easier to port across multiple operating systems, which may have different thread APIs, or no thread support at all. (For even stronger arguments along these same lines, see John Ousterhout's presentation.)

Longer answer: More than one thread can still use this code, if only one thread runs the library code proper, and the other thread(s) communicate with the library only by setting global 'flag' variables. (For one possible use of this technique, see the answer to this question.)

Another possible way to access the code from multiple threads is to have each thread use its own "UsageEnvironment" and "TaskScheduler" objects, and thus its own event loop. The objects created by each thread (i.e., using its own "UsageEnvironment") must not interact (except via global variables).

However, when using the "LIVE555 Streaming Media" code, you should be familiar with event-driven programming, and understand that an event-driven application can perform at least as well as one that uses threads (unless you're actually running on a multiprocessor, in which case it's usually simpler to have your application consist of multiple processes (not just threads) - one running on each processor). Note, in particular, that you do not need multiple threads in order to transmit (or receive) multiple streams concurrently.

When I use a receiver application (that uses the "LIVE555 Streaming Media" code) to receive an incoming RTP/UDP (or raw-UDP) stream, I see significant network packet loss. Can anything be done to improve this?

First, you should make sure that your network has sufficient bandwidth for your data stream.

However, packet loss can also be caused by insufficiently large socket reception buffers in the receiver's operating system. By default, the "LIVE555 Streaming Media" code asks the operating system to allocate at least 50 kBytes of buffer memory for each incoming datagram socket. (Note the call to increaseReceiveBufferTo() in "liveMedia/MultiFramedRTPSource.cpp".) However, you can also ask to increase this buffer size, by calling increaseReceiveBufferTo() again, within your own application code. (Note, for example, the code for the VLC media player, which sets the OS's internal receive buffer to 2,000,000 bytes for the video stream, and 100,000 bytes for the audio stream - see "modules/demux/live555.cpp".)

Note: If you are using the Linux OS, you might first need to set the OS's maximum buffer size, before you call em>increaseReceiveBufferTo(). You do this by running (as root):

    sysctl net.core.rmem_max = the_maximum_buffer_size_that_you_need

Is there any more documentation for these libraries?

For now, the best way to understand how to use the libraries is to (i) study the example programs in the "testProgs" directory, (ii) study the library code itself, and (iii) ask questions on the "live-devel" mailing list. (You may also find the 'Doxygen' source code documentation - in particular, the "Medium" class hierarchy - useful.)

Unfortunately there's no additional documentation right now (more documentation is on the "to do" list, but features requested by paying consulting clients take priority).

I would like to see new feature X added to the code. How soon can you do this for me?

The highest-priority features are those that have been requested by paying consulting clients. If your company is interested in providing funding for the development of a particular feature, please email "support(at)live555.com".

The "test*Streamer" test programs read from a file. Can I modify them so that they take input from a MPEG encoder instead, so I can stream live (rather than prerecorded) video and/or audio?

Yes. The easiest way to do this is to change the appropriate "test*Streamer.cpp" file to read from "stdin" (instead of "test.*"), and then pipe the output of your encoder to (your modified) "test*Streamer" application. (Even simpler, if your operating system represents the MPEG input device as a file, then you can just use the name of this file (instead of "test.*").)

Alternatively, if your encoder presents you with a sequence of frames, rather than a sequence of bytes, then a more efficient solution would be to write your own "FramedSource" subclass that encapsulates your encoder, and delivers audio or video frames directly to the appropriate "*RTPSink" object. This avoids the need for an intermediate 'framer' filter that parses the input byte stream. (If, however, you are streaming MPEG-4 or MPEG-2 video with "B" frames, then you should include the appropriate "*DiscreteFramer" filter, in order to generate correct presentation times.)

For a model of how to do that, see "liveMedia/DeviceSource.cpp" (and "liveMedia/include/DeviceSource.hh"). You will need to fill in parts of this code to do the actual read from your encoder.

But what about the "testOnDemandRTSPServer" test program (for streaming via unicast)? How can I modify it so that it takes input from a live source instead of from a file?

First, you will need to modify "testProgs/testOnDemandRTSPServer.cpp" to set the variable "reuseFirstSource" to "True". This tells the server to use the same input source object, even if more than one client is streaming from the server concurrently.

Then, as before, if your input device is accessible by a file name (including "stdin" for standard input), then simply replace the appropriate "test.*" file name with the file name of your input device.

If, however, you have written your own "FramedSource" subclass (e.g., based on "DeviceSource") to encapsulate your input source, then the solution is a little more complicated. In this case, you will also need to define and implement your own new subclass of "OnDemandServerMediaSubsession" that gets its input from your live source, rather than from a file. In particular, you will need to provide your own implementation of the two pure virtual functions "createNewStreamSource()" and "createNewRTPSink()". For a model of how to do this, see the existing "FileServerMediaSubsession" subclass that is used to stream your desired type of data from an input file. Note that:

Your "createNewStreamSource()" implementation will create (and return) an instance of your input source object. (You should also set the "estBitrate" result parameter to be the estimated bit rate (in kbps) of the stream. This estimate is used to determine the frequency of RTCP packets; it is not essential that it be accurate.)
Your "createNewRTPSink()" implementation will create (and return) an appropriate new "RTPSink" (subclass) object. (The code for this will usually be the same as the code for "createNewRTPSink()" in the corresponding "FileServerMediaSubsession" subclass.)

Can the RTSP server implementation (e.g., as demonstrated by the "testOnDemandRTSPServer" test program) stream to set-top boxes (STBs)?

Yes, our RTSP server implementation can, in principle, stream to RTSP-compliant STBs. In practice, however, there are some issues to note:

Many (most?) STBs do not support RTP; instead, they handle streams of raw-UDP packets. There is currently no standard way to use the RTSP protocol to request raw-UDP streams, so different STB (and server) manufacturers tend to define their own (often incompatible) RTSP extensions for this.
STBs typically handle only MPEG Transport Streams, not Elementary Streams or other data formats. (Because MPEG Transport Streams contain their own timestamps, they can be streamed via raw-UDP, with RTP timestamps not being required.)
Some STBs do not handle multicast streams.

However, our RTSP server implementation (and, in particular, the "LIVE555 Media Server" and the "testOnDemandRTSPServer" demo application) can stream MPEG Transport Stream data (with 'trick play' support) to Amino STBs (in particular, the AmiNet model 103 and 110). Note that, for this to work, the STB's RTSP client software must be configured to use "nCube" mode (the default?), not "Oracle" or "Mediabase" mode. Also (as noted above), the input source (to the RTSP server) must be a MPEG-2 Transport Stream.

Does the RTSP implementation (client and/or server) support 'trick mode' operations (i.e., seek, fast-forward, reverse play)?

When talking about "trick mode support", it's important to distinguish between RTSP client support, and RTSP server support.

Our RTSP client implementation fully supports 'trick play' operations. Note the "start", "end" and "scale" parameters to "RTSPClient::playMediaS(ubS)ession".

Our RTSP server implementation also supports 'trick play' operations, but note that parts of this are (necessarily) media type specific. I.e., there has to be some new code added for each different type of media file that we wish to stream. This functionality has already been provided for some types of media file.

To add 'trick play' support for a media type (that does not already support it), changes need to be made to the corresponding subclass of "ServerMediaSubsession":

To add support for seeking within a stream, you will need to implement the following virtual functions:
- ```
virtual float duration() const;
```
  Returns the file's duration, in seconds
- ```
virtual void seekStreamSource(FramedSource* inputSource, float seekNPT);
```
  (Attempts to) seek within the input source.
To add support for 'fast forward' and/or 'reverse play', you will also need to implement the following virtual functions:
- ```
virtual void testScaleFactor(float& scale);
```
  Inspects the input value of "scale", and, if necessary, changes it to a nearby value that we support. (E.g., if the input value of "scale" is 3.3, you might change it to 3 (an integer).) If there's no 'nearby' value that you support, just set "scale" to 1 (the default value).
- ```
virtual void setStreamSourceScale(FramedSource* inputSource, float scale);
```
  Actually sets the scale factor for a specific input source. (The value of "scale" will previously have been passed in and out of "testScaleFactor()", so we know that it is a value that we support.)

The "testStreamer" and "testReceiver" test programs use multicast. Can I modify them to use unicast instead?

Yes, you can do this, but you should first convince yourself that this is something that you really want to do. If you're streaming over a LAN, then you should continue to use multicast - it's simpler, and allows more than one receiver to access the stream, without data duplication. The only time you should consider using unicast is if you are streaming over a wider-area network that does not support multicast routing. (Note also that the RTSP server that's built in to the "test*Streamer" programs does not work with unicast streams. To play unicast streams from a RTSP server, you should instead use the existing "testOnDemandRTSPServer" test program as a model. This is usually better than trying to modify one of the "test*Streamer" applications.)

If you still wish to change the "test*Streamer" programs to stream using unicast, then do the following:

In "test*Streamer.cpp", change "destinationAddressStr" to the (unicast) IP address of the intended destination.
In the corresponding "test*Receiver.cpp", change "sessionAddressStr" to "0.0.0.0".
(optional) If you also want to send RTCP packets (e.g., RTCP Receiver Reports) back to the streaming server, then you will also need to do the following - in "test*Receiver.cpp" - after you've created "rtcpGroupsock". (In this example, suppose that the streaming server has IP address "10.20.30.40" and uses port 6667 for RTCP.):
```
    struct in_addr serverAddress;
    serverAddress.s_addr = our_inet_addr("10.20.30.40");
    rtcpGroupsock.changeDestinationParameters(serverAddress, 6667, 255);
```

For many of the "test*Streamer" test prpgrams, the built-in RTSP server is optional (and disabled by default). For "testAMRudioStreamer", "testMPEG4VideoStreamer" and "testWAVAudioStreamer", however, the built-in RTSP server is manditory. Why?

For those media types (AMR audio, MPEG-4 video, and PCM audio, respectively), the stream includes some codec-specific parameters that are communicated to clients out-of-band, in a SDP description. Because these parameters - and thus the SDP description - can vary from stream to stream, the only effective way to communicate this SDP description to clients is using the standard RTSP protocol. Therefore, the RTSP server is a manditory part of these test programs.

Where can I find an example of a MPEG-4 Elementary Stream video file that I can use (as "test.m4e") in the "testMPEG4VideoStreamer" or "testOnDemandRTSPServer" demo applications?

One easy way to get a MPEG-4 Video Elementary Stream file is to find a public MPEG-4 RTSP/RTP stream, and then run "openRTSP" on it.

If you search in Google for

+"rtsp://" +".mp4"

then you'll find several "rtsp://" URLs that contain MPEG-4 video content. You can try receiving some of these using "openRTSP" (add the "-t" option if you're behind a firewall).

This should give you two files: "video-MP4V-ES-1" and "audio-MPEG4-GENERIC-2". (If, instead, you get a file "video-H264-1", then this is H.264 video, not MPEG-4 video. Try again with another stream.) Rename the file "video-MP4V-ES-1" as "test.m4e", and you will be able to use it in "testMPEG4VideoStreamer" and "testOnDemandRTSPServer".

We have also made some example MPEG-4 Video Elementary Stream (".m4e") files available online here.

Can I stream H.264 video via RTP? None of the test programs illustrate this.

Yes, the library supports both the transmission (using "H264VideoRTPSink") and reception (using "H264VideoRTPSource") of H.264 RTP streams. The only reason why there's no test program for this is that there doesn't seem to be a single, well-defined way to package H.264 Elementary Stream video data in byte-stream form. Instead, most developers who wish to stream H.264 video will have a (hardware or software) encoder that delivers discrete NAL units, rather than an unstructured byte stream.

If you have an encoder that delivers H.264 NAL units, you must first write your own "H264VideoStreamFramer" subclass that encapsulates it. You can then feed this to a "H264VideoRTPSink", to stream the H.264 stream via RTP.

How can I stream JPEG video via RTP? There is no demo application for this.

See here and here.

When I ran "testMPEG1or2VideoStreamer", I saw several error messages like "saw unexpected code 0x000001e0". What's wrong?

By default, "testMPEG1or2VideoStreamer" assumes that its input is a MPEG (1 or 2) Video Elementary Stream - i.e., a stream that consists only of MPEG video. Your input is probably instead a MPEG Program Stream - a stream that consists of both video and audio, multiplexed together. You can play this stream by uncommenting the line

    #define SOURCE_IS_PROGRAM_STREAM 1

in "testMPEG1or2VideoStreamer.cpp". Alternatively, you could run "testMPEG1or2AudioVideoStreamer" instead of "testMPEG1or2VideoStreamer" (and thereby stream audio as well as video).

When I stream a MP3 file (using "testMP3Streamer" or "testOnDemandRTSPServer"), I find that QuickTime Player will not play the stream. What's wrong?

This is a known (and longstanding) bug in QuickTime Player: It cannot play MP3 audio RTP streams. (It will play MP3 files OK, and will play MPEG layer I or layer II audio RTP streams - but not MPEG layer III (i.e., MP3) RTP streams.)

Blame Apple for this. They have known about this bug for many years, but - for some odd reason - do not consider it a high priority bug.

Instead, we recommend that you use the VLC media player.

When I try to receive a stream using the "openRTSP" command-line client, the RTSP protocol exchange appears to work OK, but the resulting data file(s) are empty. What's wrong?

RTP/UDP media (audio and/or video) packets from the server are not reaching the client, most likely because there is a firewall somewhere inbetween that is blocking UDP packets. (Note that the RTSP protocol uses TCP, not UDP.) To correct this, either fix your firewall, or else request RTP-over-TCP streaming, using the "-t" option to "openRTSP".

The calls to "doEventLoop()" in the test programs do not return. How can I arrange for "doEventLoop" to return - e.g., if the user clicks on a "stop" button in a GUI?

"TaskScheduler::doEventLoop()" takes an optional "watchVariable" parameter that can be used for this purpose. See the definition of "TaskScheduler::doEventLoop()" in the file "UsageEnvironment/include/UsageEnvironment.hh". (Also, if you use the event loop's "watchVariable" parameter, then note this point.)

I tried using one of the test programs to stream my file, but it didn't work. Why?

First, are you sure that your file is of the correct type? (For example, if you are using "testMPEG1or2VideoStreamer", then your input file ("test.mpg") must be a MPEG Video Elementary Stream file.)

If you're sure that your file is of the correct type, then please put it on a publically-accessible web (or FTP) server, and post the URL (not the file itself) to the "live-devel" mailing list, and we'll take a look at it, to see if we can figure out what's wrong.

The test programs worked OK for me, but then I modified one of them, and it no longer works. What's wrong?

Since we don't know what modifications you made, we can't tell :-) But remember: You have complete source code! You began with one of the test programs - code that already works - and then you modified it. Therefore, you should have all the information that you need to figure out what's wrong with your program. (Of course, if you find a genuine bug in the LIVE555 Streaming Media code, then please post it to "live-devel" mailing list.)

I tried to play a "rtsp://" URL (using VLC, MPlayer or openRTSP), or a "sip:" URL (using playSIP), but I got an error message "RTP payload format unknown or not supported". Why?

The problem here is that the "liveMedia" library does not support the "RTP payload format" that is used to stream data with this particular codec.

An "RTP payload format" for a codec is a set of rules that define how the codec's media frames are packed within RTP packets. This is usually defined by an IETF RFC, or - for newer payload formats - an IETF Internet-Draft. However, a few RTP payload formats (usually those whose MIME subtype begins with "X-") are proprietary, are not defined in publically-available documents.

The "liveMedia" library supports many, but not all, RTP payload formats. If you encounter a RTP payload format that is not supported, but which is defined by a publically-available document, then we may be able to add support for it, if there is sufficient interest.

Why do most RTP sessions use separate streams for audio and video? How can a receiving client synchronize these streams?

Sending audio and video in separate RTP streams provides a great deal of flexibility. For example, this makes it possible for a player to receive only the audio stream, but not video (or vice-versa). It would even be possible to have one computer receive and play audio, and a separate computer receive and play video.

These audio and video streams are synchronized using RTCP "Sender Report" (SR) packets - which map each stream's RTP timestamp to 'wall clock' (NTP) time. For more information, see the IETF's RTP/RTCP specification.

Receivers can then use this mapping to synchronize the incoming RTP streams. The LIVE555 Streaming Media code does this automatically: For subclasses of "RTPSource", the "presentationTime" parameter that's passed to the 'afterGettingFunc' of "getNextFrame()" (see "liveMedia/include/FramedSource.hh") will be an accurate, time-synchronized time. (For this to work, you need to have also created a "RTCPInstance" for each RTP source.)

For example, if you use "openRTSP" to receive RTSP/RTP streams, then the contents of each RTP stream (audio and video) are written into separate files. This is done using the "FileSink" class. If you look at the "FileSink::afterGettingFrame()" member function, you'll notice that there's a "presentationTime" parameter for each incoming frame. Some other receiver could use the "presentationTime" parameter to synchronize audio and video.

I have a general question about RTP/RTCP, RTSP, or SIP - not specifically related to the LIVE555 Streaming Media software. Where can I get more information?

RTP/RTCP is standardized by the IETF's Audio/Video Transport ("avt") working group. In particular, note the RTP/RTCP specification.

RTSP is standardized by the IETF's Multiparty Multimedia Session Control ("mmusic") working group. (For more information, see www.rtsp.org)

SIP is standardized by the IETF's Session Initiation Protocol ("sip") and Session Initiation Proposal Investigation ("sipping") working groups. (For more information, see Henning Schulzrinne's site.)

Live Networks, Inc. (LIVE555.COM)