mirror of
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
synced 2024-08-26 02:39:48 +00:00
![Laurent Pinchart](/assets/img/avatar_default.png)
Drivers often need to associate pipeline objects to entities, and to take stream state into account when configuring entities and links. The pipeline API helps drivers manage that information. When starting streaming, drivers call media_entity_pipeline_start(). The function marks all entities connected to the given entity through enabled links, either directly or indirectly, as streaming. Similarly, when stopping the stream, drivers call media_entity_pipeline_stop(). The media_entity_pipeline_start() function takes a pointer to a media pipeline and stores it in every entity in the graph. Drivers should embed the media_pipeline structure in higher-level pipeline structures and can then access the pipeline through the media_entity structure. Link configuration will fail with -EBUSY by default if either end of the link is a streaming entity, unless the link is marked with the MEDIA_LNK_FL_DYNAMIC flag. Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com> Acked-by: Hans Verkuil <hverkuil@xs4all.nl> Signed-off-by: Mauro Carvalho Chehab <mchehab@redhat.com>
353 lines
14 KiB
Text
353 lines
14 KiB
Text
Linux kernel media framework
|
|
============================
|
|
|
|
This document describes the Linux kernel media framework, its data structures,
|
|
functions and their usage.
|
|
|
|
|
|
Introduction
|
|
------------
|
|
|
|
The media controller API is documented in DocBook format in
|
|
Documentation/DocBook/v4l/media-controller.xml. This document will focus on
|
|
the kernel-side implementation of the media framework.
|
|
|
|
|
|
Abstract media device model
|
|
---------------------------
|
|
|
|
Discovering a device internal topology, and configuring it at runtime, is one
|
|
of the goals of the media framework. To achieve this, hardware devices are
|
|
modeled as an oriented graph of building blocks called entities connected
|
|
through pads.
|
|
|
|
An entity is a basic media hardware building block. It can correspond to
|
|
a large variety of logical blocks such as physical hardware devices
|
|
(CMOS sensor for instance), logical hardware devices (a building block
|
|
in a System-on-Chip image processing pipeline), DMA channels or physical
|
|
connectors.
|
|
|
|
A pad is a connection endpoint through which an entity can interact with
|
|
other entities. Data (not restricted to video) produced by an entity
|
|
flows from the entity's output to one or more entity inputs. Pads should
|
|
not be confused with physical pins at chip boundaries.
|
|
|
|
A link is a point-to-point oriented connection between two pads, either
|
|
on the same entity or on different entities. Data flows from a source
|
|
pad to a sink pad.
|
|
|
|
|
|
Media device
|
|
------------
|
|
|
|
A media device is represented by a struct media_device instance, defined in
|
|
include/media/media-device.h. Allocation of the structure is handled by the
|
|
media device driver, usually by embedding the media_device instance in a
|
|
larger driver-specific structure.
|
|
|
|
Drivers register media device instances by calling
|
|
|
|
media_device_register(struct media_device *mdev);
|
|
|
|
The caller is responsible for initializing the media_device structure before
|
|
registration. The following fields must be set:
|
|
|
|
- dev must point to the parent device (usually a pci_dev, usb_interface or
|
|
platform_device instance).
|
|
|
|
- model must be filled with the device model name as a NUL-terminated UTF-8
|
|
string. The device/model revision must not be stored in this field.
|
|
|
|
The following fields are optional:
|
|
|
|
- serial is a unique serial number stored as a NUL-terminated ASCII string.
|
|
The field is big enough to store a GUID in text form. If the hardware
|
|
doesn't provide a unique serial number this field must be left empty.
|
|
|
|
- bus_info represents the location of the device in the system as a
|
|
NUL-terminated ASCII string. For PCI/PCIe devices bus_info must be set to
|
|
"PCI:" (or "PCIe:") followed by the value of pci_name(). For USB devices,
|
|
the usb_make_path() function must be used. This field is used by
|
|
applications to distinguish between otherwise identical devices that don't
|
|
provide a serial number.
|
|
|
|
- hw_revision is the hardware device revision in a driver-specific format.
|
|
When possible the revision should be formatted with the KERNEL_VERSION
|
|
macro.
|
|
|
|
- driver_version is formatted with the KERNEL_VERSION macro. The version
|
|
minor must be incremented when new features are added to the userspace API
|
|
without breaking binary compatibility. The version major must be
|
|
incremented when binary compatibility is broken.
|
|
|
|
Upon successful registration a character device named media[0-9]+ is created.
|
|
The device major and minor numbers are dynamic. The model name is exported as
|
|
a sysfs attribute.
|
|
|
|
Drivers unregister media device instances by calling
|
|
|
|
media_device_unregister(struct media_device *mdev);
|
|
|
|
Unregistering a media device that hasn't been registered is *NOT* safe.
|
|
|
|
|
|
Entities, pads and links
|
|
------------------------
|
|
|
|
- Entities
|
|
|
|
Entities are represented by a struct media_entity instance, defined in
|
|
include/media/media-entity.h. The structure is usually embedded into a
|
|
higher-level structure, such as a v4l2_subdev or video_device instance,
|
|
although drivers can allocate entities directly.
|
|
|
|
Drivers initialize entities by calling
|
|
|
|
media_entity_init(struct media_entity *entity, u16 num_pads,
|
|
struct media_pad *pads, u16 extra_links);
|
|
|
|
The media_entity name, type, flags, revision and group_id fields can be
|
|
initialized before or after calling media_entity_init. Entities embedded in
|
|
higher-level standard structures can have some of those fields set by the
|
|
higher-level framework.
|
|
|
|
As the number of pads is known in advance, the pads array is not allocated
|
|
dynamically but is managed by the entity driver. Most drivers will embed the
|
|
pads array in a driver-specific structure, avoiding dynamic allocation.
|
|
|
|
Drivers must set the direction of every pad in the pads array before calling
|
|
media_entity_init. The function will initialize the other pads fields.
|
|
|
|
Unlike the number of pads, the total number of links isn't always known in
|
|
advance by the entity driver. As an initial estimate, media_entity_init
|
|
pre-allocates a number of links equal to the number of pads plus an optional
|
|
number of extra links. The links array will be reallocated if it grows beyond
|
|
the initial estimate.
|
|
|
|
Drivers register entities with a media device by calling
|
|
|
|
media_device_register_entity(struct media_device *mdev,
|
|
struct media_entity *entity);
|
|
|
|
Entities are identified by a unique positive integer ID. Drivers can provide an
|
|
ID by filling the media_entity id field prior to registration, or request the
|
|
media controller framework to assign an ID automatically. Drivers that provide
|
|
IDs manually must ensure that all IDs are unique. IDs are not guaranteed to be
|
|
contiguous even when they are all assigned automatically by the framework.
|
|
|
|
Drivers unregister entities by calling
|
|
|
|
media_device_unregister_entity(struct media_entity *entity);
|
|
|
|
Unregistering an entity will not change the IDs of the other entities, and the
|
|
ID will never be reused for a newly registered entity.
|
|
|
|
When a media device is unregistered, all its entities are unregistered
|
|
automatically. No manual entities unregistration is then required.
|
|
|
|
Drivers free resources associated with an entity by calling
|
|
|
|
media_entity_cleanup(struct media_entity *entity);
|
|
|
|
This function must be called during the cleanup phase after unregistering the
|
|
entity. Note that the media_entity instance itself must be freed explicitly by
|
|
the driver if required.
|
|
|
|
Entities have flags that describe the entity capabilities and state.
|
|
|
|
MEDIA_ENT_FL_DEFAULT indicates the default entity for a given type.
|
|
This can be used to report the default audio and video devices or the
|
|
default camera sensor.
|
|
|
|
Logical entity groups can be defined by setting the group ID of all member
|
|
entities to the same non-zero value. An entity group serves no purpose in the
|
|
kernel, but is reported to userspace during entities enumeration. The group_id
|
|
field belongs to the media device driver and must not by touched by entity
|
|
drivers.
|
|
|
|
Media device drivers should define groups if several entities are logically
|
|
bound together. Example usages include reporting
|
|
|
|
- ALSA, VBI and video nodes that carry the same media stream
|
|
- lens and flash controllers associated with a sensor
|
|
|
|
- Pads
|
|
|
|
Pads are represented by a struct media_pad instance, defined in
|
|
include/media/media-entity.h. Each entity stores its pads in a pads array
|
|
managed by the entity driver. Drivers usually embed the array in a
|
|
driver-specific structure.
|
|
|
|
Pads are identified by their entity and their 0-based index in the pads array.
|
|
Both information are stored in the media_pad structure, making the media_pad
|
|
pointer the canonical way to store and pass link references.
|
|
|
|
Pads have flags that describe the pad capabilities and state.
|
|
|
|
MEDIA_PAD_FL_SINK indicates that the pad supports sinking data.
|
|
MEDIA_PAD_FL_SOURCE indicates that the pad supports sourcing data.
|
|
|
|
One and only one of MEDIA_PAD_FL_SINK and MEDIA_PAD_FL_SOURCE must be set for
|
|
each pad.
|
|
|
|
- Links
|
|
|
|
Links are represented by a struct media_link instance, defined in
|
|
include/media/media-entity.h. Each entity stores all links originating at or
|
|
targetting any of its pads in a links array. A given link is thus stored
|
|
twice, once in the source entity and once in the target entity. The array is
|
|
pre-allocated and grows dynamically as needed.
|
|
|
|
Drivers create links by calling
|
|
|
|
media_entity_create_link(struct media_entity *source, u16 source_pad,
|
|
struct media_entity *sink, u16 sink_pad,
|
|
u32 flags);
|
|
|
|
An entry in the link array of each entity is allocated and stores pointers
|
|
to source and sink pads.
|
|
|
|
Links have flags that describe the link capabilities and state.
|
|
|
|
MEDIA_LNK_FL_ENABLED indicates that the link is enabled and can be used
|
|
to transfer media data. When two or more links target a sink pad, only
|
|
one of them can be enabled at a time.
|
|
MEDIA_LNK_FL_IMMUTABLE indicates that the link enabled state can't be
|
|
modified at runtime. If MEDIA_LNK_FL_IMMUTABLE is set, then
|
|
MEDIA_LNK_FL_ENABLED must also be set since an immutable link is always
|
|
enabled.
|
|
|
|
|
|
Graph traversal
|
|
---------------
|
|
|
|
The media framework provides APIs to iterate over entities in a graph.
|
|
|
|
To iterate over all entities belonging to a media device, drivers can use the
|
|
media_device_for_each_entity macro, defined in include/media/media-device.h.
|
|
|
|
struct media_entity *entity;
|
|
|
|
media_device_for_each_entity(entity, mdev) {
|
|
/* entity will point to each entity in turn */
|
|
...
|
|
}
|
|
|
|
Drivers might also need to iterate over all entities in a graph that can be
|
|
reached only through enabled links starting at a given entity. The media
|
|
framework provides a depth-first graph traversal API for that purpose.
|
|
|
|
Note that graphs with cycles (whether directed or undirected) are *NOT*
|
|
supported by the graph traversal API. To prevent infinite loops, the graph
|
|
traversal code limits the maximum depth to MEDIA_ENTITY_ENUM_MAX_DEPTH,
|
|
currently defined as 16.
|
|
|
|
Drivers initiate a graph traversal by calling
|
|
|
|
media_entity_graph_walk_start(struct media_entity_graph *graph,
|
|
struct media_entity *entity);
|
|
|
|
The graph structure, provided by the caller, is initialized to start graph
|
|
traversal at the given entity.
|
|
|
|
Drivers can then retrieve the next entity by calling
|
|
|
|
media_entity_graph_walk_next(struct media_entity_graph *graph);
|
|
|
|
When the graph traversal is complete the function will return NULL.
|
|
|
|
Graph traversal can be interrupted at any moment. No cleanup function call is
|
|
required and the graph structure can be freed normally.
|
|
|
|
Helper functions can be used to find a link between two given pads, or a pad
|
|
connected to another pad through an enabled link
|
|
|
|
media_entity_find_link(struct media_pad *source,
|
|
struct media_pad *sink);
|
|
|
|
media_entity_remote_source(struct media_pad *pad);
|
|
|
|
Refer to the kerneldoc documentation for more information.
|
|
|
|
|
|
Use count and power handling
|
|
----------------------------
|
|
|
|
Due to the wide differences between drivers regarding power management needs,
|
|
the media controller does not implement power management. However, the
|
|
media_entity structure includes a use_count field that media drivers can use to
|
|
track the number of users of every entity for power management needs.
|
|
|
|
The use_count field is owned by media drivers and must not be touched by entity
|
|
drivers. Access to the field must be protected by the media device graph_mutex
|
|
lock.
|
|
|
|
|
|
Links setup
|
|
-----------
|
|
|
|
Link properties can be modified at runtime by calling
|
|
|
|
media_entity_setup_link(struct media_link *link, u32 flags);
|
|
|
|
The flags argument contains the requested new link flags.
|
|
|
|
The only configurable property is the ENABLED link flag to enable/disable a
|
|
link. Links marked with the IMMUTABLE link flag can not be enabled or disabled.
|
|
|
|
When a link is enabled or disabled, the media framework calls the
|
|
link_setup operation for the two entities at the source and sink of the link,
|
|
in that order. If the second link_setup call fails, another link_setup call is
|
|
made on the first entity to restore the original link flags.
|
|
|
|
Media device drivers can be notified of link setup operations by setting the
|
|
media_device::link_notify pointer to a callback function. If provided, the
|
|
notification callback will be called before enabling and after disabling
|
|
links.
|
|
|
|
Entity drivers must implement the link_setup operation if any of their links
|
|
is non-immutable. The operation must either configure the hardware or store
|
|
the configuration information to be applied later.
|
|
|
|
Link configuration must not have any side effect on other links. If an enabled
|
|
link at a sink pad prevents another link at the same pad from being disabled,
|
|
the link_setup operation must return -EBUSY and can't implicitly disable the
|
|
first enabled link.
|
|
|
|
|
|
Pipelines and media streams
|
|
---------------------------
|
|
|
|
When starting streaming, drivers must notify all entities in the pipeline to
|
|
prevent link states from being modified during streaming by calling
|
|
|
|
media_entity_pipeline_start(struct media_entity *entity,
|
|
struct media_pipeline *pipe);
|
|
|
|
The function will mark all entities connected to the given entity through
|
|
enabled links, either directly or indirectly, as streaming.
|
|
|
|
The media_pipeline instance pointed to by the pipe argument will be stored in
|
|
every entity in the pipeline. Drivers should embed the media_pipeline structure
|
|
in higher-level pipeline structures and can then access the pipeline through
|
|
the media_entity pipe field.
|
|
|
|
Calls to media_entity_pipeline_start() can be nested. The pipeline pointer must
|
|
be identical for all nested calls to the function.
|
|
|
|
When stopping the stream, drivers must notify the entities with
|
|
|
|
media_entity_pipeline_stop(struct media_entity *entity);
|
|
|
|
If multiple calls to media_entity_pipeline_start() have been made the same
|
|
number of media_entity_pipeline_stop() calls are required to stop streaming. The
|
|
media_entity pipe field is reset to NULL on the last nested stop call.
|
|
|
|
Link configuration will fail with -EBUSY by default if either end of the link is
|
|
a streaming entity. Links that can be modified while streaming must be marked
|
|
with the MEDIA_LNK_FL_DYNAMIC flag.
|
|
|
|
If other operations need to be disallowed on streaming entities (such as
|
|
changing entities configuration parameters) drivers can explictly check the
|
|
media_entity stream_count field to find out if an entity is streaming. This
|
|
operation must be done with the media_device graph_mutex held.
|