3DPlanning

From Spice

APITraceRuns

Summary:

  1. Write linux guest DRM driver
  2. Write windows guest WDDM driver
  3. Server side rendering
  4. Client side rendering
Plan for Adding support for 3D guest API's in Spice
===================================================

This is a work in progress. Actually, it's the plan for the work in progress. Please feel free to criticize anything and everything.

Objective: support GL and Direct3D applications.
Sub objectives:
 Support Aero and gnome-shell/unity/compositing managers.
 Use WDDM in windows guests.
 Use direct rendering in linux guests.

Considered and abandoned direction:
 use VirtualBox approach: support only GL in the windows driver, use direct3d dlls courtesy of wine.
 Advantages: already has a proof of concept and it's got a compatible (same) license.
 Disadvantages: experimental (VBox 4.1.2) and replaces system provided dlls.

Approach taken: same as VMWare vmwgfx
 reuse mesa for both windows and linux driver.
 store per context state for client reconnection.
 TGSI intermediate form. Server only deals with TGSI, doesn't know about GL or Direct3D. Only recourse to GL is when it wants to render, and that will be in a shared server and client module (same as usage of pixman for 2d today)
 Advantages:
  Single state machine to handle for state saving for client reconnections.
  Future proof for adding guest and client API.
  TGSI is superset of supported guest APIs - no loss of data (this should be verified..)
 Disadvantages:
  windows-windows and linux-linux require two conversions. Conversions are not expensive.

Since there are now more then one person working on this, namely me, Arnon and possibly Yonit, the plan is to start towards a linux on linux demo while at the same time starting work on a WDDM driver.

The stages planned for linux support:

 1. Add a kernel drm driver and use it in qxl_drv
  - no mesa yet.
  - but keep mesa requirements in mind.
  - handle allocations by the kernel. we currently use mspace.
  Results:
   support for interrupts in linux guest.
   kernel drm module for qxl.
   move qxl command pushing / reading (command, cursor and release ring) to kernel?

 2. Add gallium support for spice. This is the largest and yet to be subdivided stage:
  - add qxl winsys
  - add qxl pipe driver
  - reuse existing mesa and gallium components
  - add qxl commands required by mesa
   buffer object create / destroy
  - add spice server rendering of TGSI to GL
  Specifically this stage will not result in remoting of 3D commands, but only of images.
  No protocol changes.
  No requirement to store per surface state.
  Results:
   streaming (server rendering) 3D remoting. (like VirtualGL)
   no problem to connect and disconnect a client since no client side 3D state exposed.
   client unchanged (no new protocol. specifically very bad bandwidth since we will send individual compressed frames to not force lossy mjpeg compression)

 3. Add 3D protocol and client side rendering.
  Specify 3D protocol
  Do not render in the server unless required (GetPixels)
  Implement surface state saving <- TBD. Initial approach: save everything since the last SwapBuffers (equivalent TGSI operation).
  Results:
   
Other notes to consider:
 impact of switching qxl to virtio (possible, not planned yet):
  rings not relevant. memory allocation - we should support guest non contiguous memory from the start.
 video passthrough:
  both linux and windows have APIs for that. For windows having a WDDM driver is a realistic requirement, and it is part of the plan for 3d support. For linux mesa/gallium also support VA so this will also be an enabler.

In parallel a WDDM driver will be developed. This will be completely
parallel, will only depend on completion of the server and qemu parts of
stage 2 for server side rendering, and stage 3 is guest independent. A
stage 4 would consist of TGSI to Direct3D for better windows-windows
support.

High level diagrams for the 3 stages follow, sources attached (using asciio)

C - changed
N - new

                Stage 1         
        Add DRM                 

          .------------------.
          |       App        |
          |                  |
          |   .--------.     |
          |   | libX11 |     |
          |   '--------'     |
          '--------|---------'
                   v
      .-------------------------.
      |          Xorg           |
      |                         |
      | .CCCCCCCCCCCCCCCCCCCCC. |
      | |     qxl_drv.so      | |
      | | changes: use libdrm | |
      | '---------------------' |
      |                         |
      |    .NNNNNNNNNNNNNNN.    |
      |    |  libdrm_qxl   |    |
      |    '---------------'    |
      '------------|------------'
                   |
                   v
     .---------------------------.
     |          kernel           |
     |                           |
     |    .NNNNNNNNNNNNNNN.      |
     |    |    qxl.ko     |      |
     |    '---------------'      |
     '------------|--------------'
                  |
                  v
      .-----------------------.
      |         qemu          |
      |                       |
      |  .------------------. |
      |  .       qxl        . |
      |  '------------------' |
      |            |          |
      '------------|----------'
                   |
                   v
        .---------------------.
        |       server        |
        |                     |
        '---------------------'



                                 Stage 2                   
              3D Server side: DRI, Pipe, Winsys, TGSI to GL

            .--------------------------------------------------------.
            |                          App                           |
            |                                                        |
            | .--------.   .--------.    .NNNNNNNNN.  .-------------.|
            | | libX11 |   | libGL  |--->| qxl_dri |->| libdrm_qxl  ||
            | '--------'   '--------'    '---------'  '-------------'|
            '------|-------------------------------------------------'
                   |
                   |
                   v
      .-------------------------.    .----------------------------------.
      |          Xorg           |    |             qxl_dri              |
      |                         |    |                                  |
      |  .-------------------.  |    |  .-------------------.           |
      |  |    qxl_drv.so     |  |    |  |   Mesa GL core    |           |
      |  '-------------------'  |    |  '-------------------'           |
      |                         |    |                                  |
      |     .CCCCCCCCCCCCC.     |    |  .-----------------------------. |
      |     | libdrm_qxl  |     |    |  | Mesa->Gallium state tracker | |
      |     '-------------'     |    |  '-----------------------------' |
      |            |            |    |                                  |
      '------------|------------'    |  .NNNNNNNNNNNNNNNNNNNNNNNNNN.    |
                   |                 |  | Gallium->QXL pipe driver |    |
                   v                 |  '--------------------------'    |
     .---------------------------.   |                                  |
     |          kernel           |   |  .NNNNNNNNNNNNNNNNNNNNN.         |
     |                           |   |  | Gallium->QXL winsys |         |
     |    .---------------.      |   |  '---------------------'         |
     |    |    qxl.ko     |      |   |                                  |
     |    '---------------'      |   '----------------------------------'
     '------------|--------------'
                  |
                  |
                  v                  .-----------------------------.
   .---------------------------.     |           server            |
   |           qemu            |     |                             |
   |                           |     |   Add surface type          |
   |   .CCCCCCCCCCCCCCCCC.     |     |                             |
   |   .       qxl       .     |     |  .NNNNNNNNNNNNNNNNNNNNN.    |
   |   . new messages    .---------->|  |     TGSI to GL      |    |
   |   . in qxl_dev.h    .     |     |  |                     |    |
   |   '-----------------'     |     |  '---------------------'    |
   |                           |     |                             |
   '---------------------------'     '-----------------------------'



                                 Stage 3                   
              Protocol, Persistence, Client side rendering 

            .--------------------------------------------------------.
            |                          App                           |
            |                                                        |
            | .--------.   .--------.    .---------.  .-------------.|
            | | libX11 |   | libGL  |--->| qxl_dri |->| libdrm_qxl  ||
            | '--------'   '--------'    '---------'  '-------------'|
            '------|-------------------------------------------------'
                   v
      .-------------------------.
      |          Xorg           |
      |  .-------------------.  |
      |  |    qxl_drv.so     |  |        .---------------------------.
      |  '-------------------'  |        |          kernel           |
      |     .-------------.     |        |    .---------------.      |
      |     | libdrm_qxl  |------------->|    |    qxl.ko     |      |
      |     '-------------'     |        |    '---------------'      |
      |                         |        '------------|--------------'
      '-------------------------'                     v
                                        .---------------------------.
                                        |           qemu            |
 .----------------------------------.   |                           |
 |              server              |   |  .----------------------. |
 |                                  |   |  .         qxl          . |
 |     .---------------------.      |   |  '----------------------' |
 |     |     TGSI to GL      |      |   '--------------|------------'
 |     '---------------------'      |                  |
 |                                  |<-----------------'
 |   .NNNNNNNNNNNNNNNNNNNNNNNNNNN.  |
 |   | Per surface/context state |  |
 |   '---------------------------'  |
 '----------------------------------'
                   |
                   v
      .NNNNNNNNNNNNNNNNNNNNNNNN.
      |      3D Protocol       |
      | TGSI as protocol       |
      '------------------------'
                   |
                   v
  .---------------------------------.
  |             client              |
  |                                 |
  |    .NNNNNNNNNNNNNNNNNNNNN.      |
  |    |     TGSI to GL      |      |
  |    '---------------------'      |
  '---------------------------------'