[MPlayer-dev-eng] Libvo2 draft

[David Holm]

Ok, I've read through the documentation and here are my thoughts...

>
>This is a brief description on libvo2 interface. It is not C code, just
>draft scheme. Feel free to suggest exact parameters.
>I have tried to put some numbering. So if you want to reply then put the topic number in the subject line. Please don't reply to the whole draft, or at least don't include big paragraphs from it.
>I'm gonna put this text as attachment to force you to copy only the parts you want to reply;)
>
>   Best Regards
>Ivan Kalvachev
>
>1.libvo2 drivers 
>1.1 functions
>Currently these functions are implemented:
>  init
>  control
>  start
>  stop
>  get_surface
>  flip_image
>
>They are simple enough. So I introduce to be implemented and these functions:
>  query
>  update_surface - renamed draw
>  hw_decode
>  subpicture
>
>Here is detailed description of new functions:
>
>  query - the negotiation is more complex than just finding which imgfmt the device could show, we must have list of capabilities, testing modes, etc. this function will have at least 3 modes:
>    a) return list of available modes with description.
>    b) check could we use this mode with these parameter. E.g. if we want RGB32 with 3 surfaces for windows image 800x600 we may get out of video memory. We don't want error because this mode could be used with 2 surfaces.
>    c) return supported subpicture formats if any
>
>
>
>As you may see I have removed some functionality from control() and made separate function. Why? It is generally good thing functions that are critical to the driver to have it's own implementation. 
>
Personally I'd prefer to have this in control, but the concept is good 
and easily implemented with a struct the same way the handle is (with 
memset(...,0,...) to be able to extend it without breaking old devices).
I could implement this easily, all I need is Arpi's approval.

>
>
>  update_surface - as in the note above, this is draw function. Why I change it's name? I have 2 reasons, first I don't want implementation like vo1, second it really must update video surface, it must directly call the system function that will do it. This function should work only with slices, the size of slice should not be limited and should be passed (e.g ystart, yend), if we want draw function, we will call one form libvo2 core, that will call this one with start=0; ymax=Ymax;. Also some system screen update functions wait for vertical retrace before return, other functions just can't handle partial updates. In this case we should inform libvo2 core that device cannot slice, and libvo2 core must take care of the additional buffering.
>
well, as it is now slices are implemented in a way such that it is 
always a complete frame because it keeps the data from the previous call 
and changes only the data within the slice.
Are there hardware devices that support this?? if so, we just fetch it 
using query (or control, whitchever gets chosen)

>
>
>  hw_decode - to make all dvb,dxr3, tv etc. developers happy. This function is for you. Be careful, don't OBSEBE it, think and for the future, this function should have and ability to control HW IDCT, MC that one day will be supported and under linux. Be careful:)
>
This sounds unneeded, check my example vo2_dxr3 implementation, I 
believe the current functions are sufficient, surface returns a memory 
area to draw into and flip calls whatever functions are required by 
hardware, I really see no need for this function at all.
Future features are to be controlled by control(..), that's one of it's 
purposes...

>
>
>  subpicture - this function will place subtitles. It must be called once to place them and once to remove them, it should not be called on every frame, the driver will take care of this. 
>    Currently I propose this implementation: we get array of bitmaps. Each one have its own starting x, y and it's own height and width, each one (or all together) could be in specific imgfmt (spfmt). THE BITMAPS SHOULD NOT OVERLAP! This may not be hw limitation but sw subtitles may get confused if they work as 'c' filter (look my libvo2 core).
>
I think this is also unnecessary, it's better this is handled by the 
libvo2 core depending on the VO2CTRL_OVERLAY control. We want to get 
away from device developers having to do stuff themselves the core might 
as well do. If really needed I prefer implementing something like 
VO2CTRL_OVERLAY_CALLBACK and passing it a callback function as parameter.

>
>    
>1.2 control()
>OK, here is list of some control()s that I think that could be useful:
>    SET_ASPECT
>    SET_SCALLE_X, SET_SIZE_X
>    SET_SCALLE_Y, SET_SIZE_Y
>    RESET_SIZE
>    GET/SET_POSITION_X
>    GET/SET_POSTIION_Y
>    GET/SET_RESOLUTION
>    GET/SET_DISPLAY
>    GET/SET_ATTRIBUTES
>
>Here is description of how these controls to be used:
>
>  SET_ASPECT - this is the move/video aspect, why not calculate it in different place (mplayer.c) and pass the results to driver by set_size_x/y. First this is only if hardware could scale. Second we may need this value if we have TV and we won't calculate new height and width.
>
>  SET_SCALLE_X/Y - this is to enlarge/downscale  the image, it WILL NOT override SET_ASPECT, they will have cumulative effect, this could be used for deinterlacing (HALF SIZE). Second if we want to zoom 200% we don't want to lose aspect calculations. Or better SET_SCALLE to work with current size?
>
>  SET_SIZE_X/Y - This is for custom enlarge, to save some scale calculation and for more precise results.
>
>  RESET_SIZE - Set the original size of image, we must call SET_ASPECT agein.
>
>  GET/SET_POSOTION_X/Y - This if for windows only, to allow custom move on window.
>
>  GET/SET_RESOLUTION - change resolution and/or bpp if possible. To be used for window or if we want to change the given resolution of the current fullscreen mode (NOT TO SET IT just to change it if we don't like it)
>
>  GET/SET_DISPLAY - mainly for X11 and remote displays. Not very useful, but may be handy.
>
>  GET/SET_ATTRIBUTES - Xv overlays have contrast, brightness, hue, saturation etc. these and others could be controlled by this. If we want to query it we must call GET_*, and the to check does our attribute is in there (xv developers be careful, 2 or 3 of default attributes sometimes are not queried by X, but could be set).
>
Then why not expand this to GET/SET_BRIGHTNESS, HUE, SATURATION?? easier 
to handle. The dxr3 also has these abilities...
The other controls seems useful to me and I see no reason not to 
implement them

>
>
>Do you think that TV encodings (NTSC,PAL,SECAM) should have it's own attribute?
>I would like to hear the GUI developers. Could we separate Mouse/Keyboard from the driver. What info do you need to do it. Don't forget that SDL have it's own keyboard/mouse interface.
>Maybe we should allow video driver to change the libin driver ? 
>
or maybe this is a great idea for libvo3?

>
>
>1.3. query()
>Here come and some attributes for the queried modes, each supported mode should have such description.
>It is even possible to have more than one mode that could display given imgfmt.
>{
>  Scale y/n  - hardware scale, do you think that we mast have one for x and one for y (win does)?
>
>  Fullscreen y/n - if the supported mode is fullscreen, if we have yv12 for fullscreen and window we must threat them as separate modes.
>Window y/n - same as Fullscreen.
>
>  GetSurface y/n - if driver could give us video surface we'll use get_surface()
>
>  UpdateSurfece y/n - if driver will update video surface through sys function (X,SDL)
>
>  HWdecode y/n  - if driver could take advantage of hw_decode()
>
>  MaxSurfaces 1..n - Theoretical maximum of surfaces
>
>  SubPicture y/n - Could we put subpicture (OSD) of any kind by hw
>
>  WriteCombine y/n - if GetSurface==yes, most (or all) pci&agp cards are extremely slow on byte access, this is hint to vo2 core those surfaces that got affected by WC. This is only a hint.
>
>  us_clip y/n - if UpdateSurface=yes, this shows could update_surface() remove strides (when stride> width ), this is used and for cropping. If not, we must do it.
>
>  us_slice y/n - if UpdateSurface=yes, this shows that after executing update_surface(), the function won't wait for vertical retrace, and we could update surface slice by slice. If us_slice=0 we will have to accumulate all slices in one buffer.
>
>  us_upsidedown - if UpdateSufrace=yes, this shows that update_suface() could flip  the image vertically. In some case this could be united with us_clip /stride game/
>
>  switch_resoliton y/n - if window=y, this shows could we switch resolution of desktop, if fullscreen=y, shows that we could change resolution, after we have set the fullscreen mode. 
>
>  deinterlace y/n - indicates that the device could deinterlace on it's own (radeon, TV).
>
>1.4 conclusion
>As you see, I have removed all additional buffering from the driver. There is a lot of functionality should be checked and handled by libvo2 core.
>First we should check what else could be added to this draft. Then to check all cases and how to handle them.
>Some of the parameters should be able to be overriden by user config, mainly to disable buggy modes or parameters. I belive that this should not be done by command line as there are enough commands now.
>
>//---------------------------
>2. libvo2 core
>2.1 functions
>now these function are implemented:
>    init
>    new
>    start
>    query_format
>    close
>
>and as draw.c:
>    choose_buffering
>    draw_slice_start
>    draw_slice
>    draw_frame
>    flip
>
>init() is called at mplayer start. internal initialisation.
>new() -> rename to open_drv() or something like this.
>query_format -> not usable in this form, this function mean that all negotiation will be performed outside libvo2. Replace or find better name. 
>close -> open/close :)
>
>choose_buffering - all buffering must stay hidden. The only exception is for hw_decode. In the new implementation this functions is not usable.
>draw_slice_start, draw_slice -> if you like it this way, then it's OK.
>draw_frame -> classic draw function.
>
>2.2 Minimal buffering
>I should say that I stand after the idea all buffering, postprocessing, format conversion , sw draw of subtitles, etc to be done in libvo2 core. Why? First this is the only way we could fully control buffering and decrease it to minimum. Less buffers means less coping. In some cases this could have the opposite effect (mpeg2 with internal buffers in video memory without HW IDCT, MC, or if we have unaligned write).
>The first step of the analyse is to find out what we need:
>
>DECODER   -   type_of_buffer:{internal/static/normal}, 
>              slice:{not/supported}
>
>FILTER 1..x - processing:{ c-copy(buff1,buff2), p-process(buff1) }, 
>              slice:{not/supported}
>              write_combine:{not/safe}, 
>              runtime_remove:{static/dynamic}
>
>VIDEO_OUT  -  method:{get_surface/update_surface}, 
>              slice:{not/supported}, 
>              write_combine:{not/safe},
>              clip:{can/not},
>              upsidedown:(can/not),
>              surfaces:{1/2/3,..,n}
>
>  If we want direct rendering we need normal buffer, no filters, and (at least) 2 video surfaces. (we may allow 'p' filter like subtitles).
>  If we have static buffer, we have 2 choices: to render in 1 surface (visual flickering) or to make additional buffering and draw on flip_page (like in libvo1).
>
>Here I introduce and one letter codes that I use for analyse.
>Details: 
>
>DECODER - We always get buffer from the decoder, some decoders could give pointer to it's internal buffers, other takes pointers to buffers where they should store the final image. Some decoders could call draw_slice after they have finished with some portion of the image.
>  type_of_buffer - I take this from the current libvo2 spec.  I call 'I' internal buffer (readonly), 'K' static buffer(one,constant pointer), and 'B' - normal buffer. 
>  slice - this flag shows that decoder knows and want to work with slices.
>
>FILTER - postprocessing, sw drawing subtitles, format conversion, crop,  additional filters.
>  slice - could this filter work with slice order. We could use slice even when decoder does not support slice, we just need 2 or more filters that does. This could give us remarkable speed boost.
>  processing - some filters can copy the image from one buffer to the other, I call them 'c', convert and crop(stride copy) are good examples but don't forget simple 1:1 copy. Other filters does process only part if the image, and could reuse the given buffer, e.g. putting subtitles. Other filters could work in one buffer, but could work and with 2, I call them 't' class, after analyse they will fade to 'c' or 'p'. 
>  runtime_remove - postprocess with autoq. Subtitles appear and disappear, should we copy image from one buffer to another if there is no processing at all?
>
>//clip, crop, upsidedown - all 'c' filters must support strides, and should be able to remove them and to make some tricks like crop and upside_down.
>
>VIDEO_OUT - take a look of libvo2 driver I propose.
>  method - If we get surface -'S'. If we use draw* (update_surface) - 'd'
>
>As you may see hd_decode don't have complicated buffering:)
>I make the analyse this way. First I put decoder buffer, then I put all filters, that may be needed, and finally I put video out method.
>
I'm working on a vo2_context struct which will set the appropriate 
function for the current device (i.e. pixel_converter, filters etc) I'll 
extend it with the things in your list that are missing...

>
>
>2.3. Rules for minimal buffering
>The rules are these: 
>The 'p' filters process in the buffer of the left, if we have 'I' buffer then insert copy and new 'B' buffer. 
>With 'c' filter we must make sure that we have buffer ('B' or 'S') from the right(->) side. 
>We must take care that, if we have S we need to finish with copy ('c'), and if we have 'd' we must end with some kind of buffer.
>In the usual case 't' are replaced with 'p' except when 't' is before 'S'.
>If we have 'B S' or 'K S' we may make direct rendering and remove the 'B' or 'K' buffer.
>We must have at least one 'c' if we have to make crop, clip, or flip image upside down.
>Take care for the additional buffering when we have 1 surface (the libvo1 way).
>Be aware that some filters must be before other. E.g. Postporcessing should be before subtitles:)
>If we want scale (-zoom), and vo2 driver can't make it then add and scale filter 'c'. For better understanding I have one convert filter that can copy, convert, convert and scale. The only thing that is missing now is simple scale (yv12).
>
>I have made grammar for these cases but it is too big and I hope that something cleverer could be made. Don't think that having only 3 filters (postporcess, convert/copy, subtitles) may make the things simpler. This algorithm could be used and for numerous filters used in encoders.
>
>2.4 Negotiation
>Few words about negotiation. It is hard thing to find the best mode. Here is algorithm that could find the best mode. But first I must say that we need some kind of weight for the filters and drawing. I think that we could use something like megabytes/second, something that we may measure or benchmark.
>  1. We choose codec
>  2. We choose video driver.
>  3. For each combination find the total weight and if there are any optional filters find min and max weight. Be careful max weight is not always at maximum filters!!
>  4. Compare the results.
>I may say that we don't need automatic codec selection as now we could put best codecs at beginning of codecs.conf as it is now. We may need to make same thing with videodrv.conf :)
>
Sounds like a good plan too me...

Thanks for getting involved in the libvo2 development, and perfect 
timing btw ;) Since I've just begun working fulltime on libvo2. All I 
need now is Arpi's feedback on this, make modifications and send up the 
second draft for headers, after that it's coding time...

//David Holm

>