From 5b4e6c33c80702e09f46c2497c33d1299a8d3453 Mon Sep 17 00:00:00 2001 From: Wim Taymans Date: Sat, 8 Oct 2005 16:49:15 +0000 Subject: [PATCH] docs/design/: Many doc updates. Original commit message from CVS: * docs/design/part-TODO.txt: * docs/design/part-clocks.txt: * docs/design/part-events.txt: * docs/design/part-gstbin.txt: * docs/design/part-gstelement.txt: * docs/design/part-gstpipeline.txt: * docs/design/part-live-source.txt: * docs/design/part-messages.txt: * docs/design/part-overview.txt: * docs/design/part-states.txt: Many doc updates. --- ChangeLog | 14 ++ docs/design/part-TODO.txt | 13 +- docs/design/part-clocks.txt | 15 ++- docs/design/part-events.txt | 47 +++++-- docs/design/part-gstbin.txt | 21 ++- docs/design/part-gstelement.txt | 7 +- docs/design/part-gstpipeline.txt | 16 ++- docs/design/part-live-source.txt | 2 +- docs/design/part-messages.txt | 21 ++- docs/design/part-overview.txt | 37 +++--- docs/design/part-states.txt | 213 ++++++++++++++++++++++++++----- 11 files changed, 316 insertions(+), 90 deletions(-) diff --git a/ChangeLog b/ChangeLog index 3c50be0e4..f5474df60 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,3 +1,17 @@ +2005-10-08 Wim Taymans + + * docs/design/part-TODO.txt: + * docs/design/part-clocks.txt: + * docs/design/part-events.txt: + * docs/design/part-gstbin.txt: + * docs/design/part-gstelement.txt: + * docs/design/part-gstpipeline.txt: + * docs/design/part-live-source.txt: + * docs/design/part-messages.txt: + * docs/design/part-overview.txt: + * docs/design/part-states.txt: + Many doc updates. + 2005-10-08 Wim Taymans * gst/gstevent.c: diff --git a/docs/design/part-TODO.txt b/docs/design/part-TODO.txt index 107b6dea9..799b334e0 100644 --- a/docs/design/part-TODO.txt +++ b/docs/design/part-TODO.txt @@ -10,10 +10,19 @@ - unlinking pads in the PAUSED state needs to make sure the stream thread is not executing code. Can this be done with a flush to unlock all downstream chain - functions? + functions? Do we do this automatically or let the app handle this? -- implement clock selection as explained in part-gstpipeline.txt +- implement clock selection as explained in part-gstpipeline.txt. - when a pipeline with a live source goes to PAUSED again, a sample is prerolled in the sinks. This sample should be discarded, possibly with a flush event started from the source. + +- implement latency calculation for live sources. + +- implement master/slave clocks. + +- implement QOS. + +- implement BUFFERSIZE. + diff --git a/docs/design/part-clocks.txt b/docs/design/part-clocks.txt index f8cfc94a3..9c70aaf9e 100644 --- a/docs/design/part-clocks.txt +++ b/docs/design/part-clocks.txt @@ -5,8 +5,8 @@ To synchronize the different elements, the GstPipeline is responsible for selecting and distributing a global GstClock for all the elements in it. This selection happens whenever an element is added or removed from the -pipeline. Whever the clock changes in a pipeline, a message is posted on -the bus signaling the new clock to the application. +pipeline. Whever the clock changes in a pipeline, a NEW_CLOCK message is +posted on the bus signaling the new clock to the application. The GstClock returns a monotonically increasing time with the method _get_time(). Its accuracy and base time depends on the specific clock @@ -39,22 +39,22 @@ GstClock absolute time. Timestamps ---------- -The combination of the last DISCONT event and the buffer timestamps +The combination of the last NEWSEGMENT event and the buffer timestamps express the presentation stream time of the buffer. The stream time of a buffer is calculated as follows: ST = TS - DT where: TS = buffer timestamp - DT = DISCONT timestamp + DT = NEWSEGMENT timestamp ST = buffer stream time The reason for not making the buffer times express the stream time directly is for the following reasons: - demuxers are easier if they can just copy the timestamps as encoded in - the file. The initial discont event would contain the lowest timestamp in + the file. The initial NEWSEGMENT event would contain the lowest timestamp in the stream which makes the stream time start from 0. - pipelines requiring retimestamping of buffers can efficiently adjust - the timestamp in the discont events and have all buffers retimestamped + the timestamp in the NEWSEGMENT events and have all buffers retimestamped automatically. - resync after different kinds of seeks is easier. @@ -110,7 +110,8 @@ invalid. These clock operations do not operate on the stream time, so the callbacks will also occur when not in PLAYING state as if the clock just keeps on -running. +running. Some clocks however do not progress when the element that provided +the clock is not PLAYING. Clock implementations diff --git a/docs/design/part-events.txt b/docs/design/part-events.txt index 4e1bb6e61..ec95728be 100644 --- a/docs/design/part-events.txt +++ b/docs/design/part-events.txt @@ -16,6 +16,7 @@ Different types of events exist to implement various functionalities. GST_EVENT_NEWSEGMENT: A new group of buffers with common start time GST_EVENT_TAG: Stream metadata. GST_EVENT_FILLER: Filler for sparse data streams + GST_EVENT_BUFFERSIZE: Buffer size requirements GST_EVENT_QOS: A notification of the quality of service of the stream GST_EVENT_SEEK: A seek should be performed to a new position in the stream GST_EVENT_NAVIGATION: A navigation event. @@ -64,7 +65,7 @@ An EOS event sent on a srcpad returns GST_FLOW_UNEXPECTED. The downstream element should forward the EOS event to its downstream peer elements. This way the event will eventually reach the renderers which should -then post an EOS message on the bus. +then post an EOS message on the bus when in PLAYING. An element might want to flush its internally queued data before forwarding the EOS event downstream. This flushing can be done in the same thread as @@ -85,13 +86,15 @@ sending data. An element that sends EOS on a pad should stop sending data on that pad. Source elements typically pause() their task for that purpose. -By default, the pipeline collects all EOS events from all the sinks before -passing the EOS message to the application. +By default, a GstBin collects all EOS events from all its sinks before +posting the EOS message to its parent. The EOS is only posted on the bus by the sink elements in the PLAYING state. If the EOS event is received in the PAUSED state, it is queued until the element goes to PLAYING. +A seek event on an element flushes all pending EOS messages. + NEWSEGMENT ------------- @@ -123,7 +126,7 @@ The NEWSEGMENT event can be send from both the application and the streaming thread and should be serialized with the buffers. Buffers should be clipped within the range indicated by the newsegment event -start and stop values. Sinks are allowed to drop buffers with timestamps out +start and stop values. Sinks must to drop buffers with timestamps out of the indicated newsegment range. If a newsegment arrives at an element not preceeded by a flush event, the @@ -131,6 +134,34 @@ streamtime of the pipeline will not be reset to 0 so any element that syncs to the clock must use the stop times of the previous newsegment events to make the buffer timestamps increasing. +TAG +--- + +The tag event is sent downstream when an element has discovered metadata +tags in a media file. Encoders can use this event to adjust their tagging +system. A tag is serialized with buffers. + + +FILLER +------ + + +BUFFERSIZE +---------- + +An element can suggest a buffersize for downstream elements. This is +typically done by elements that produce data on multiple source pads +such as demuxers. + + +QOS +--- + +A QOS, or quality of service message, is generated in an element to report +to the upstream elements about the current quality of the stream. This +is typically done by the sinks that measure the amount of framedrops they +have. + SEEK ---- @@ -194,11 +225,3 @@ of a navigation event such as a mouse movement or button click. Navigation events travel upstream. -TAG ---- - -The tag event is sent downstream when an element has discovered metadata -tags in a media file. Encoders can use this event to adjust their tagging -system. A tag is serialized with buffers. - - diff --git a/docs/design/part-gstbin.txt b/docs/design/part-gstbin.txt index 78486b2e8..8c8cbe0f3 100644 --- a/docs/design/part-gstbin.txt +++ b/docs/design/part-gstbin.txt @@ -36,7 +36,6 @@ operations on itself to all of its children. This includes: - state changes - index get/set - clock gst/set - - scheduler set/get The state change distribution is the most complex and is explained in part-states.txt. @@ -50,7 +49,12 @@ messages from children. The bus for receiving messages from children is distinct from the bin's own externally-visible GstBus. Messages received from children are forwarded intact onto the bin's -external message bus, except for EOS which is handled specially. +external message bus, except for EOS and SEGMENT_START/DONE which are +handled specially. + +STATE_CHANGED messages received from the children are used to trigger a +recalculation of the current state of the bin, as described in +part-states.txt. The application can retrieve the external GstBus and integrate it in the mainloop or it can just _pop() messages off in its own thread. @@ -75,10 +79,21 @@ The list of queued EOS messages is cleared when the bin goes to PAUSED again. This means that all elements should repost the EOS message when going to PLAYING again. + +SEGMENT_START/DONE +------------------ + +* not implemented. + +A bin collects SEGMENT_START messages but does not post them to the application. +It counts the number of SEGMENT_START messages and posts a SEGMENT_STOP message +to the application when an equal number of SEGMENT_STOP messages where received. + + Subclassing ----------- Subclasses of GstBin are free to implement their own add/remove implementations. It is a good idea to update the GList of children so that the _iterate() functions -can still be used. +can still be used if the custom bin allows acces to its children. diff --git a/docs/design/part-gstelement.txt b/docs/design/part-gstelement.txt index 6266518f5..ebdd88164 100644 --- a/docs/design/part-gstelement.txt +++ b/docs/design/part-gstelement.txt @@ -70,9 +70,4 @@ Ghost Pads State ----- -Elements use state to determine what the are capable of doing at any given moment. The states are defined as follows: - -NULL No state is held for the element -READY Devices are open -PLAYING -PAUSED +An element has a state. More info in part-states.txt. diff --git a/docs/design/part-gstpipeline.txt b/docs/design/part-gstpipeline.txt index 8c6c4e04b..f16b990b7 100644 --- a/docs/design/part-gstpipeline.txt +++ b/docs/design/part-gstpipeline.txt @@ -18,15 +18,21 @@ State changes In addition to the normal state change procedure of its parent class GstBin, the pipeline performs the following actions during a state change: + - NULL -> READY: + - set the bus to non-flushing + - READY -> PAUSED: + - reset the stream time to 0 + + - PAUSED -> PLAYING: - Select and set a clock. + - calculate base time using the stream time. - PAUSED -> PLAYING: - - calculate the stream time. + - calculate the stream time when the pipeline was stopped. -The GstPipeline will also wait for any async state change to complete before -proceeding to the next state change. This is usefull for the application because -it does not have to deal with ASYNC state changes then. + - READY -> NULL: + - set the bus to flushing Clock selection @@ -65,7 +71,7 @@ When performing a seek on the pipeline element using gst_element_send_event(), the pipeline performs the following actions: - record the current state of the pipeline. - - set the pipeline to paused + - set the pipeline to paused if a FLUSHING seek is requested - send the seek event to all sinks - when a FLUSH seek is done, the stream_time is set 0 again. - restore old state of the pipeline. diff --git a/docs/design/part-live-source.txt b/docs/design/part-live-source.txt index 9283bc7a2..8696d82fa 100644 --- a/docs/design/part-live-source.txt +++ b/docs/design/part-live-source.txt @@ -20,7 +20,7 @@ the get_state() function would block on the sinks. A gstbin therefore always performs a zero timeout get_state() on its elements to discover the NO_PREROLL (and ERROR) elements before performing -a blocking wait on all elements. +a blocking wait. Scheduling diff --git a/docs/design/part-messages.txt b/docs/design/part-messages.txt index 1a50e947f..f3d5d8b77 100644 --- a/docs/design/part-messages.txt +++ b/docs/design/part-messages.txt @@ -48,13 +48,22 @@ Message types GST_MESSAGE_STATE_CHANGED: - An element changed state in the pipeline. The message carries the old an new - state of the element. + An element changed state in the pipeline. The message carries the old, new + and pending state of the element. GST_MESSAGE_STEP_DONE: An element stepping frames has finished. + GST_MESSAGE_CLOCK_PROVIDE: + + An element notifies it capability of providing a clock for the pipeline. + + GST_MESSAGE_CLOCK_LOST: + + The current clock as selected by the pipeline became unusable. The pipeline + will select a new clock on the next PLAYING state change. + GST_MESSAGE_NEW_CLOCK: A new clock was selected for the pipeline. @@ -71,12 +80,16 @@ Message types GST_MESSAGE_APPLICATION: - An element posted an application specific message. + The application posted a message. + + GST_MESSAGE_ELEMENT: + + Element-specific message, see the specific element's documentation GST_MESSAGE_SEGMENT_START: An element started playback of a new segment. This message is not forwarded - the the application but is used internally to scheduler SEGMENT_DONE messages. + the the application but is used internally to schedule SEGMENT_DONE messages. GST_MESSAGE_SEGMENT_DONE: diff --git a/docs/design/part-overview.txt b/docs/design/part-overview.txt index 735c42c4b..d953e3793 100644 --- a/docs/design/part-overview.txt +++ b/docs/design/part-overview.txt @@ -19,7 +19,7 @@ Introduction is an object that performs some action on a multimedia stream such as: - read a file - - decode or encoder between formats + - decode or encode between formats - capture from a hardware device - render to a hardware device - mix or multiplex multiple streams @@ -283,7 +283,7 @@ Pipeline clock of one GST_SECOND per second. Clock values are expressed in nanoseconds. Elements use the clock time to synchronized the playback of data. - Before the pipeline is set to PAUSED, the pipeline asks each element if they can + Before the pipeline is set to PLAYING, the pipeline asks each element if they can provide a clock. The clock is selected in the following order: - If the application selected a clock, use that one. @@ -316,13 +316,14 @@ Pipeline states All intermediate states are traversed for each element resulting in the following chain of state changes: - alsasink to READY: the audio device is opened - mp3dec to READY: the decoding library is initialized - filesrc to READY: the file is opened - alsasink to PAUSED: alsasink is a sink and returns ASYNC because it did not receive - data yet. - mp3dec to PAUSED: nothing happens - filesrc to PAUSED: a thread is started to push data to mp3dec + alsasink to READY: the audio device is probed + mp3dec to READY: nothing happens. + filesrc to READY: the file is probed + alsasink to PAUSED: the audio device is opened. alsasink is a sink and returns + ASYNC because it did not receive data yet. + mp3dec to PAUSED: the decoding library is initialized + filesrc to PAUSED: the file is opened and a thread is started to push data to + mp3dec At this point data flows from filesrc to mp3dec and alsasink. Since mp3dec is PAUSED, it accepts the data from filesrc on the sinkpad and starts decoding the compressed @@ -336,7 +337,7 @@ Pipeline states Alsasink then receives the buffer, inspects the caps and reconfigures itself to process the buffer. Since it received the first buffer of samples, it completes the state change to the PAUSED state. At this point the pipeline is prerolled and all elements have - samples. + samples. Alsasink is now also capable of providing a clock to the pipeline. Since alsasink is now in the PAUSED state it blocks while receiving the first buffer. This effectively blocks both mp3dec and filesrc in their gst_pad_push(). @@ -344,9 +345,9 @@ Pipeline states Since all elements now return SUCCESS from the gst_element_get_state() function, the pipeline can be put in the PLAYING state. - Before going to PLAYING, the pipeline samples the current time of the clock. This is - the base time. It then distributes this time to all elements. Elements can then - synchronize against the clock using the buffer timestamp+base time. + Before going to PLAYING, the pipeline select a clock and samples the current time of + the clock. This is the base time. It then distributes this time to all elements. + Elements can then synchronize against the clock using the buffer timestamp+base time. The following chain of state changes then takes place: @@ -402,8 +403,8 @@ Pipeline EOS Pipeline READY -------------- - When a running pipeline is set from the RUNNING to READY the following actions - occur in the pipeline: + When a running pipeline is set from the PLAYING to READY state, the following + actions occur in the pipeline: alsasink to PAUSED: alsasink blocks and completes the state change on the next sample. If the element was EOS, it does not wait for @@ -488,8 +489,8 @@ Pipeline seeking Since the pipeline is still PAUSED, this will preroll the next media sample in the sinks. - The last step in the seek operation is then to adjust the media time of the pipeline - to the new position and to set the pipeline back to PLAYING. + The last step in the seek operation is then to adjust the stream time of the pipeline + to 0 and to set the pipeline back to PLAYING. The sequence of events in out mp3 playback example. @@ -511,6 +512,6 @@ Pipeline seeking --------------------------> 4) FLUSH done event --------------------------> 5) NEWSEGMENT event - | e) update stream time + | e) update stream time to 0 | f) PLAY pipeline diff --git a/docs/design/part-states.txt b/docs/design/part-states.txt index 5b736fd23..a3bc7114a 100644 --- a/docs/design/part-states.txt +++ b/docs/design/part-states.txt @@ -23,6 +23,64 @@ We call the sequence NULL->PLAYING an upwards state change and PLAYING->NULL a downwards state change. +State transitions +----------------- + +the following state changes are possible: + + NULL -> READY + - The element must check if the resources it needs are available. + Audiosinks typically try to probe the device. + + READY -> PAUSED + - The element opens the device and prepares itself for PLAYING. + - the element pads are activated in order to receive data in PAUSED. + streaming threads are started. + - some elements might need to return ASYNC and complete the state change + when they have enough information. It is a requirement for sinks to + return ASYNC and complete the state change when they receive the first + buffer or EOS event (prerol). Sinks also block the dataflow when in PAUSED. + - a pipeline resets the stream time to 0. + - live sources return NO_PREROLL and don't generate data. + + PAUSED -> PLAYING + - most elements ignore this state change. + - The pipeline selects a clock and distributes this to all the children + before setting them to PLAYING. This means that it is only alowed to + synchronize on the clock in the PLAYING state. + - The pipeline uses the clock and the stream time to calculate the base time. + The base time is distributed to all children when performing the state + change. + - sink elements stop blocking on the preroll buffer or event and start + rendering the data. + - sinks can post the EOS message in the PLAYING state. It is not allowed to + post EOS when not in the PLAYING state. + - while streaming in PAUSED or PLAYING elements can create and remove + dynamic pads. + - live sources start generating data and return SUCCESS. + + PLAYING -> PAUSED + - most elements ignore this state change. + - The pipeline calculates the stream time based on the last selected clock + and the base time. It stores this information to continue playback when + going back to the PLAYING state. + - sinks unblock any clock wait calls. + - sinks return ASYNC from this state change and complete the state change + when they receive a buffer or an EOS event. + - any queued EOS messages are removed since they will be reposted when going + back to the PLAYING state. + - live sources stop generating data and return NO_PREROLL. + + PAUSED -> READY + - sinks unblock any waits in the preroll. + - elements unblock any waits on devices + - the element pads are deactivated so that streaming becomes impossible and + all streaming threads are stopped. + + READY -> NULL + - element removes any dynamically created pads + + State variables --------------- @@ -32,18 +90,17 @@ the STATE_LOCK. The STATE_LOCK protects 3 element variables: - STATE - - PENDING_STATE - - STATE_ERROR flag - - NO_PREROLL flag + - STATE_NEXT + - STATE_PENDING + - STATE_RETURN -The STATE always reflects the current state of the element. The PENDING_STATE -always reflects the required state of the element. The PENDING_STATE can be -VOID_PENDING if the element is in the right state. The STATE_ERROR flag -indicates that an error occured while doing the last state change. - -The NO_PREROLL flag indicates that the element said it was not able to preroll -in its last state change. This flag is used in live sources. +The STATE always reflects the current state of the element. +The STATE_NEXT reflects the next state the element will go to. +The STATE_PENDING always reflects the required state of the element. +The STATE_RETURN reflects the last return value of a state change. +The STATE_NEXT and STATE_PENDING can be VOID_PENDING if the element is in +the right state. Setting state on elements ------------------------- @@ -68,13 +125,16 @@ The _set_state() function can return 3 possible values: will not be able to produce data in the PAUSED state. In the case of an async state change, it is possible to proceed to the next -state before the current state change completed. After receiving an ASYNC return -value, you can use _element_get_state() to poll the status of the element. - -When setting the state of an element, the PENDING_STATE is set to the required -state and the STATE_ERROR flag is cleared. Then the state change function of the -element is called and the result of that function is used to update the STATE, -PENDING_STATE and STATE_ERROR flags. If the function returned ASYNC, this result +state before the current state change completed, however, the element will only +get to this next state before completing the previous ASYNC state change. +After receiving an ASYNC return value, you can use _element_get_state() to poll +the status of the element. If the polling returns SUCCESS, the element completed +the state change to the last requested state with _set_state(). + +When setting the state of an element, the STATE_PENDING is set to the required +state. Then the state change function of the element is called and the result of +that function is used to update the STATE and STATE_RETURN fields, STATE_NEXT, +STATE_PENDING and STATE_RETURN fields. If the function returned ASYNC, this result is immediatly returned to the caller. @@ -143,21 +203,10 @@ ASYNC, the function returns ASYNC as well. If after calling the state function on all children, one of the children returned NO_PREROLL, the function returns NO_PREROLL as well. -The current state of the bin can be retrieved with _get_state(). This function will -call the _get_state() function on all the elements. - -First the bin will perform a _get_state() on all children with a 0 timeout. This -is to find any children with an ERROR/NO_PREROLL result value. +The current state of the bin can be retrieved with _get_state(). -Then the bin performs the _get_state() with the requested timeout. The reason for -the 2 phases is that when an ERROR or NO_PREROLL result is found, a blocking -wait on the sinks might never return. - -The _get_state() function will be called on the children with the same timout value -so the function can potentially block timeout*num_children. - -The bin also updates its state variables after polling its children, this means that -the state variables of the bin are only updated after calling _get_state() on the bin. +If the bin is performing an ASYNC state change, it will automatically update its +current state fields when it receives state messages from the children. Implementing states in elements @@ -167,6 +216,106 @@ READY ----- - +upward state change +------------------- + +Upward state changes always return ASYNC either if the STATE_PENDING is +reached or not. + +Element: + + A -> B => SUCCESS + - commit state + + A -> B => ASYNC + - no commit state + - element commits state ASYNC + + A -> B while ASYNC + - update STATE_PENDING state + - no commit state + - no change_state called on element + +Bin: + + A->B: all elements SUCCESS + - commit state + + A->B: some elements ASYNC + - no commit state + - listen for commit messages on bus + - for each commit message, poll elements + - if no ASYNC elements, commit state, continue state change + to STATE_PENDING + +downward state change +---------------------- + +Downward state changes only return ASYNC if the final state is ASYNC. +This is to make sure that it's not needed to wait for an element to +complete the preroll or other ASYNC state changes when one only wants to +shut down an element. + +Element: + + A -> B => SUCCESS + - commit state + + A -> B => ASYNC not final state + - commit state on behalf of element + + A -> B => ASYNC final state + - element will commit ASYNC + +Bin: + + A -> B -> SUCCESS + - commit state + + A -> B -> ASYNC not final state + - commit state on behalf of element, continue state change + + A -> B => ASYNC final state + - no commit state + - listen for commit messages on bus + - for each commit message, poll elements + - if no ASYNC elements, commit state + + +Locking overview (element) +-------------------------- + + set_state(element) change_state (element) stream_thread commit_state (element) + + | | | | + | | | | + STATE_LOCK | | | + | | | | + |------------------------>| | | + | | | | + | | | | + | | (start_task) | | + | | | | + | | STREAM_LOCK | + | | | | + |<------------------------| | | + | ASYNC | | + STATE_UNLOCK | | + | .....sync........ STATE_LOCK | + ASYNC |----------------->| + | | + | |---> post_message(ASYNC) + | |---> if (!final) change_state (next) + | | else SIGNAL + |<-----------------| + STATE_UNLOCK + | + STREAM_UNLOCK + + + + + + -- 2.39.2