Building a Xen aware Nvidia grafic card driver: 6800GS SLI in Xen under OpenSuse10.2 64Bit

Xen is a highly professional Open Source virtualization technology which promises the least performance loss. Heres how I got the Closed Source Nvidia driver going unde OpenSuse-Xen 64bit AMD X2.

3D acceleration and SLI mode in particular are only supported by the closed source nvidia drivers. However, Xen requires a modified "Xen-Aware" kernel to run in a pravirtualized environment. Only the most recent CPUs support (SVM for AMD)fully virtulized virtual mashines that run without kernel modifications. On many websites you'll find the infromation that blah blah blah X2 CPUs from AMD would support full virtualization (SVM) blah blah. But this is wrong. Don't be mistaken only the AM2 version of X2 processors support SVM. If its not AM2 its not SVM. So don't make the same mistake like me and get a socket 939 CPU.

The Nvidia driver does not support Xen Kernels and its closed source, so you can't modifie it yourself. But not everything is lost. The nvidia driver uses a kernel module as a compatibility layer to load the binary driver. This kernel module is open source, so in theory it should be possible to modify the kernel module so that it works with Xen. And belive it or not some brave guys really hacked the kernel module to support xen-kernels.

So I had to get the hypervisor and the default-xen kernel of opoen suse 10.2 running. This is pretty easy you can use Yast to do the installation stuff (isntall hypervisor, install default-xen and have it change the bootloader settings) And of course you'll need all the devel packeges to compile the kernels.

Now boot with the default kernel and install install the binary nvidia drivers you are going to patch later.

sh NVIDIA-Linux-x86-1.0-9746-pkg2.run --x-module-path=/usr/lib64/xorg/modules/ -q

The version of the compatibility layer and the binaries have to match. the nvidia install script will do the job for you. However, it will refuse to run in a paravirualized environment. So for the Xen-kernel, be prepared to do everything manually.

Boot into the dom0 domain. you'll only have the console available because the xserver won't work. without the propreitary nvidiea driver with the GPUs in SLI mode.

init 3 # stops Xserver and swiches to runlevel 3

make cloneconfig
make prepare
wget NVIDIA_xen-9746.patch.gz
wget http://us.download.nvidia.com/XFree8...-9746-pkg1.run
sh NVIDIA-Linux-x86-1.0-9746-pkg2.run -x # extract sources
cd NVIDIA-Linux-x86-1.0-9746-pkg2/usr/src/nv
zcat ../../../../NVIDIA_xen-9746.patch.gz | patch -p4 #patch sources

Sombody stated you have to modify usr/src/nv/os-interface.c line 1288 modified to:

#if 0
to avoid compiling erroers. but this was not true for me. The driver compiled fine without any modifications. So first try without modification.

make SYSSRC=/usr/src/linux module # make nvidia kernel module
cp nvidia.ko /lib/modules/`uname -r`/updates/nvidia.ko
depmod -a
modprobe nvidia
init 5  # when successful enter runlevel 5 to start x server

Now the driver should load.
I also read this command to install the modules:
install -D -o root -g root -m 0644 nvidia.ko /lib/modules/`uname -r`/updates/nvidia.ko
But for me copying to the /udate folder of the kernel module folder was sufficient.
However my 3D performace was poor (1fps with glxgears)
So i'll try some older patches and see what happens.
The other patch I tried did not any better. It turned out they dont support SLI. If you turn of SLI by
nvidia-xconfig --no-sli
3D work descent (12800 fps for glxgears) which is about the same than without Xen.
Links to Nvidia kernel module Xen patches
another Nvidia xen patch howto for open suse

Do you want to read on <<click

Enable SLI in Linux: My Xorg.conf as an example

The SLI mode was a little harder to enable. Here is how i got it working

Install the driver according to the nvidia howto.

sax2 -r -m 0=nvidia doesent work with SLI setup

use

sax2 -r -c0 -m0=nvidia

instead. assuming that your disply is hooked up to card 0. Else you can use Card 1 but card 1 will not work for SLI

Now here was that catch: when I wanted to enable SLI GPU 0 as detectd by

nvidia-settings

needs to be the same as chip0 as reported by

sax2 -p

the can be identified by ther pci adress. And the display has to be hooked up to card 0 too. else the display will be scrambled when you starup the display.

I had to change the Init Disply first to PEG(slot2) to get this accomplished.

then you type

nvidiea-xconfig --sli=on

and your set. It si imortant that the frst available Xscreen is the screen with you monitor hooked up to it if have more that one configured in you xorg.config.

Heres the sections relevant to xscreen config of xorg.conf as a reference with sli enabeled:

Section "Module"
Load         "dbe"
Load         "type1"
Load         "freetype"
Load         "extmod"
Load         "glx"
Load         "v4l"
EndSection


Section "Monitor"
Option       "CalcAlgorithm" "XServerPool"
DisplaySize  338 270
HorizSync    30-82
Identifier   "Monitor[0]"
ModelName    "ACER AL1714"
Option       "DPMS"
VendorName   "ACR"
VertRefresh  43-75
UseModes     "Modes[0]"
EndSection

Section "Device"
BoardName    "GeForce 6800 GS"
BusID        "4:0:0"
Driver       "nvidia"
Identifier   "Device[0]"
VendorName   "NVIDIA"
EndSection


Section "Screen"
DefaultDepth 24
SubSection "Display"
  Depth      15
  Modes      "1280x1024" "1152x864" "1024x768" "800x600"
EndSubSection
SubSection "Display"
  Depth      16
  Modes      "1280x1024" "1152x864" "1024x768" "800x600"
EndSubSection
SubSection "Display"
  Depth      24
  Modes      "1280x1024" "1152x864" "1024x768" "800x600"
EndSubSection
SubSection "Display"
  Depth      8
  Modes      "1280x1024" "1152x864" "1024x768" "800x600"
EndSubSection
Device       "Device[0]"
Identifier   "Screen[0]"
Monitor      "Monitor[0]"
Option         "SLI" "on"
EndSection


Section "ServerLayout"
Identifier   "Layout[all]"
InputDevice  "Keyboard[0]" "CoreKeyboard"
InputDevice  "Mouse[1]" "CorePointer"
Screen       "Screen[0]"
EndSection


Section "DRI"
  Group      "video"
  Mode       0660
EndSection

Nvidie Linux Driver README

Do you want to read on <<click

Quake 4: enable SLI in linux

Quake 4 also runs faster in SLI mode in Linux. However, it needs to configured properly to increase performance. Here is my Quake starup script with SMP enableled.

#!/bin/sh
# Needed to make symlinks/shortcuts work.
# the binaries must run with correct working directory
cd "/usr/local/games/quake4"
export __GL_DOOM3=1
export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:."
exec ./quake4smp.x86 "$@"
export __GL_DOOM3=1

important is to add export __GL_DOOM3=1 to improve SLI performance

Do you want to read on <<click

Solved: nForce4 CK804 (Intel8x0) ALSA audiodriver uses only on two speakers for stereo playback when 5.1 6 channel speakers are connected

I have a 5.1 surround speaker set connected to my K8N ultra SLI mainboard with a CK804 sound on board. When playing MP3 I realized that Linux used only the front speaker for stereo playback, which is pretty annoying since the front speakers are not full range an desperately need the woofer to run for a nice sound. Here I write how I solved this issue.



It turned out that this issue is not a driver issue, however. This is due to a, in my opinion, odd behavior of the ALSA drivers. ALSA feeds the channel from the source to the corresponding speaker. Since you have only to two channels in stereo sound alsa sends only sound to the front (left and right) speaker. Although this behavior is very logical its verry confusing for a easyly confused noob like me and it took me quite a while to fix this problem.

First I had to make sure that the hardware is set up properly. Therefore I used a little helper applet that comes with the ALSA package:

# speaker-test -c 6 -r 48000

Now I heard sound (noise) on all speakers. If you hear something then the driver and ALSA should work.

Getting a driver update from nvidia is not necessary since they only distribute only sata and lan drivers for their nForce chip set. They included their driver sources for their on board sound to the ALSA project so it should be fine to use their driver. The CK804 is supported as as intel8x0. Anyway, here are the sites for the nvidia downloads:

http://www.nvidia.com/object/linux_nforce_1.21.html
http://http.download.nvidia.com/XFree86/nforce/1.0-0310/ReleaseNotes.html

But I did an ALSA driver update. However, this update is optional I guess.

---snip---

To do this you need to install autoconf, kernel-source, gnu make, kernel-headers with yast if not done yet.

To compile and install the latest drivers from source, first download the CVS
(Remark: devfs is depreciated in Suse 10.2 since it uses udev so we don't need to execute ./snddevices script here)

# rsync -avz --delete rsync://alsa.alsa-project.org/hg your_directory
# ./cvscompile --with-cards=intel8x0 --with-sequencer=yes
# make install
# modprobe snd-intel8x0;modprobe snd-pcm-oss;modprobe snd-mixer-oss;modprobe snd-seq-oss

---snip----

The most important step is to edit the .asoundrc. The .asoundrc is a configuration file which is read in by default and lives in your home directory.

# vi /home/xxx/.asoundrc

vi is an editor but you can use whichever you you prfer to perorm this task. xxx denotes the name of your home directory. Then add the the following code to your .asoundrc :

pcm.ch51dup {
type route
slave.pcm surround51
slave.channels 6
ttable.0.0 1
ttable.1.1 1
ttable.0.2 1
ttable.1.3 1
ttable.0.4 0.5
ttable.1.4 0.5
ttable.0.5 0.5
ttable.1.5 0.5
}

This will duplicate stereo channels to the sourround speakers but you have to use ch51dup as device for output of stereo files. In Amarok I use the configuration shown in the screenshot:

I used the following links as source:

• alsa-project home site
• Information about the alsa Module intel8x0 which is the relevant driver for the nForce CK804
• Additional Wiki page of the ALSA project about the Intel8xo module for the nForce CK80x
• Playing stereo on surround sound setup with alsa
• general infromation about the alsa .asoundrc configuration file

Technorati Tags: linux, nforce3, sound, stereo, surround, intel8x0, NVidia CK804

Do you want to read on <<click

Testing the SLI mode in Linux

If you have installed the Nvidia Drivers for Linux you want to know if the SLI mode is running. To do this you can enable the SLI Heads-Up-Display options Nvidia X Sever Settings GUI which can be installed by Yast. To do this start the utility select "OpenGL settings and check "enable sli heads up dispaly"

Now Select "X Server Display Configuration" and click on "Save to X Configuration File".

Now start an Application that should use SLI and on the left an HUD marked SLI should appear.

this should show you the overhead of you GPU but here you see no GPU load becaus the applicatiion does not put much strain on such a setup, however if you run DOOM3 or Quak4 which can both be installed on Linux you will the green bars filling up the white lines.

Do you want to read on <<click

Install Nvidia 6800GS Grafics cards in SLI mode for OpenSuse

When I first installed OpenSuse on my System it didn't set up my graphics card properly. I use a Gigabyte K8N Ultra SLI with a Nforce4 Chipset with two 6800GS in SLI mode.
Here I wrie what I've done to get my Grafic cards running. This might be also helpful for others who want to use a NVIDIA driver.
To activate the SLI mode you first have to enable the SLI mode on your mainboard by putting the SLI selector card firmly into place. Some people complain that their SLI mode doesent work and the secoind GPU is not recognised although both are installed into the Mainboard. This happens because without the SLI selector module properly in place the mainboard falls back into normal mode and thus the second PCIe_16 slot with the second SLI device is not available. In addition the SLI cards should be connected using a SLI adaptor for optimal performance.

However the setup of the GPU by Yast/Sax was not succesful so a grafical login was not possible after installation. However, It was possible to fix the Xorg.conf written by Yast using the nvidia Confugration utitlity.

First you'll have to leave the XServer using CTRL + Alt + F2 since display is messy and you can't use it. The reason is probably that Yast/Sax is configuring an Xserver on each GPU with one display for each Xserver which doesent work for me. Switch to the text console. If this fails you have to boot in debug mode with GRUB or LILO in the boot menu. For the next steps you can use the text based yast utility to do the installations.

To enable SLI mode you have to install the Closed Source Nvidia driver for OpenSuse.

Download the 64Bit Nvidia Linux Driver for the 6800GS and follow the installation instructions for Suse.

In the README they tell you to use the Sax utility to configure the Xorg.conf but this doesen't work for SLI devices I guess.

Install the nvidia-xconfig utility using Yast. This can do the modification to the /etc/X11/xorg.conf for you. Type in a root console

# nvidia-xconfig --only-one-x-screen --no-twinview --logo --sli=auto

This configures the xorg.conf to use only one display and to turn on SLI whenever posiible.

If you followed the readme of Nvidia (shutting down Xserver with Init 3 and unloading grafic driver moduls with modprobe -r then adding the repository and instlling the rpm for Suse with yast) you should now be able to start the X server with

# X

or

# Init 5

Do you want to read on <<click