|
0
|
1 % \VignetteIndexEntry{Multimedia programming with R, rdyncall, libSDL and OpenGL}
|
|
|
2 % \VignetteKeyword{low-level}
|
|
|
3 % \VignetteKeyword{rdyncall}
|
|
|
4 % \VignetteKeyword{dyncall}
|
|
|
5 % \VignetteKeyword{multimedia}
|
|
|
6 \documentclass{article}
|
|
|
7 \usepackage{url}
|
|
|
8 \begin{document}
|
|
|
9 \title{Multimedia programming with R, rdyncall, libSDL and OpenGL}
|
|
|
10 \author{Daniel Adler}
|
|
|
11 \maketitle
|
|
|
12 \section{Introduction}
|
|
|
13
|
|
|
14 Via the rdyncall package it is possible to write system-level
|
|
|
15 software in R. In this vignette we focus on multimedia programming
|
|
|
16 in R using the SDL and OpenGL shared libraries.
|
|
|
17 This vignette describes several issues involved in using the rdyncall
|
|
|
18 package with C level interfaces of OpenGL and SDL. This text is
|
|
|
19 not at all a reference to OpenGL or SDL. There are other much better
|
|
|
20 text books and tutorials available.
|
|
|
21
|
|
|
22 See \url{http://www.libsdl.org} and \url{http://www.opengl.org} for further
|
|
|
23 links to tutorials.
|
|
|
24
|
|
|
25 \section{Dynamic R bindings of shared libraries}
|
|
|
26
|
|
|
27 Initially, one needs to dynamically link SDL, OpenGL and OpenGL Utility
|
|
|
28 Library to the R session.
|
|
|
29 This can be done using the rdyncall package and the dynports.
|
|
|
30
|
|
|
31 <<>>=
|
|
|
32 library(rdyncall)
|
|
|
33 dynport(SDL)
|
|
|
34 dynport(GL)
|
|
|
35 dynport(GLU)
|
|
|
36 @
|
|
|
37
|
|
|
38 \section{Initialize SDL}
|
|
|
39
|
|
|
40 We need to initialize the SDL library using the \verb@SDL_Init()@ function
|
|
|
41 giving a bitmask of subsystems to be enabled (such as Video, Audio, Joystick etc.).
|
|
|
42
|
|
|
43 <<>>=
|
|
|
44 subsystems <- SDL_INIT_VIDEO
|
|
|
45 SDL_Init(subsystems)
|
|
|
46 @
|
|
|
47
|
|
|
48 The variable subsystem is a bitmask encoded in a double constant, also
|
|
|
49 the C API expects to have an unsigned int. rdyncall cares for a correct
|
|
|
50 coercion to unsigned int. By addition of several double constants,
|
|
|
51 we can initialize multiple subsystems.
|
|
|
52
|
|
|
53 To see what subsystems SDL offers, we can explore the SDL dynport using e.g.
|
|
|
54 the apropos base R command:
|
|
|
55
|
|
|
56 <<>>=
|
|
|
57 apropos("SDL_INIT_")
|
|
|
58 @
|
|
|
59
|
|
|
60 \section{Opening an SDL/OpenGL Video Surface}
|
|
|
61
|
|
|
62 Let us open a new OpenGL Display using \verb@SDL_SetVideoMode()@. The parameters are:
|
|
|
63
|
|
|
64 \begin{description}
|
|
|
65 \item[width] The width in pixel.
|
|
|
66 \item[height] The height in pixel
|
|
|
67 \item[bpp]
|
|
|
68 \item[flags] Bit flags to be combined with '+'. \verb@SDL_OPENGL@ and \verb@SDL_DOUBLEBUF@ are typical for OpenGL real-time applications.
|
|
|
69 \end{description}
|
|
|
70
|
|
|
71 Some useful Flags:
|
|
|
72
|
|
|
73 \begin{description}
|
|
|
74 \item[SDL\_OPENGL] Request an OpenGL context to be initialized for this surfacce.
|
|
|
75 \item[SDL\_DOUBLEBUF] Rendering into a back buffer and allow to flip the surfaces in one shot.
|
|
|
76 \item[SDL\_FULLSCREEN] Put the display in fullscreen mode.
|
|
|
77 \end{description}
|
|
|
78
|
|
|
79 <<eval=FALSE>>=
|
|
|
80 width <- 640
|
|
|
81 height <- 480
|
|
|
82 bitsperpixel <- 32
|
|
|
83 flags <- SDL_OPENGL+SDL_DOUBLEBUF
|
|
|
84 surface <- SDL_SetVideoMode(width,height,bitsperpixel,flags)
|
|
|
85 @
|
|
|
86
|
|
|
87 The surface object can be printed in R:
|
|
|
88
|
|
|
89 <<eval=FALSE>>=
|
|
|
90 print(surface)
|
|
|
91 @
|
|
|
92
|
|
|
93 \section{Clear the screen and update}
|
|
|
94
|
|
|
95 To clear the display with a constant color value, one needs to specifying the
|
|
|
96 \emph{clear color} first and then clear the \emph{color buffer}.
|
|
|
97
|
|
|
98 <<eval=FALSE>>=
|
|
|
99 glClearColor(0.1,0.2,0.3,0.0)
|
|
|
100 glClear(GL_COLOR_BUFFER_BIT)
|
|
|
101 @
|
|
|
102
|
|
|
103 To see the results, we need to flip the surface - all operations in
|
|
|
104 \emph{double buffering} graphics mode are done in the back and will
|
|
|
105 be presented in one shot to prevent semi-finished drawing artefacts.
|
|
|
106
|
|
|
107 <<eval=FALSE>>=
|
|
|
108 SDL_GL_SwapBuffers()
|
|
|
109 @
|
|
|
110
|
|
|
111 \section{Writing the mainloop}
|
|
|
112
|
|
|
113 This was interactive, but SDL and OpenGL are designed for writing
|
|
|
114 multimedia applications and we do this in R.
|
|
|
115
|
|
|
116 These types of applications run a simple loop such as the interactive
|
|
|
117 loop of R (read-evaluate-print loop).
|
|
|
118 The loop will run until the application should quit. A reference play time
|
|
|
119 will be computed for each loop iteration using \verb@SDL_GetTicks()@ (divided by 1000
|
|
|
120 gives seconds).
|
|
|
121 We will clear our display using a blinking color which changes intensity from
|
|
|
122 black to a specified color. We drop the loop after 3 seconds.
|
|
|
123
|
|
|
124 <<eval=FALSE>>=
|
|
|
125 blinkspeed <- 2.0
|
|
|
126
|
|
|
127 draw <- function(t) {
|
|
|
128 intensity <- t %% blinkspeed / blinkspeed
|
|
|
129 glClearColor(0.8*intensity,0.6*intensity,1.0*intensity,0.0)
|
|
|
130 glClear(GL_COLOR_BUFFER_BIT)
|
|
|
131 }
|
|
|
132
|
|
|
133 mainloop <- function()
|
|
|
134 {
|
|
|
135 quit <- FALSE
|
|
|
136
|
|
|
137 starttime <- SDL_GetTicks() / 1000
|
|
|
138
|
|
|
139 playtime <- 0
|
|
|
140 while(!quit) {
|
|
|
141 # blink the screen using an intensity blending from 0 to 1
|
|
|
142
|
|
|
143 draw(playtime)
|
|
|
144
|
|
|
145 SDL_GL_SwapBuffers()
|
|
|
146
|
|
|
147 # update playtime:
|
|
|
148 now <- SDL_GetTicks() / 1000.0
|
|
|
149 playtime <- now - starttime
|
|
|
150
|
|
|
151 # stop after three seconds
|
|
|
152 if (playtime > 3.0) quit <- TRUE
|
|
|
153 }
|
|
|
154 }
|
|
|
155 # run the loop
|
|
|
156 mainloop()
|
|
|
157 @
|
|
|
158
|
|
|
159 \section{Rendering 3D graphics}
|
|
|
160
|
|
|
161 To render a 3D scene, one specifies the 3D projection first, sets up the
|
|
|
162 camera position in space and then specifies the positions and primitive
|
|
|
163 geometries such as points, lines and triangles.
|
|
|
164
|
|
|
165 First we specify the attributes of our virtual camera, that is
|
|
|
166 the field of view angle, the aspect ratio between width and height (which
|
|
|
167 should match the one of our surface) and the near and far z-plane orthogonal
|
|
|
168 to the camera center projection vector.
|
|
|
169
|
|
|
170 <<eval=FALSE>>=
|
|
|
171 setupProjection <- function()
|
|
|
172 {
|
|
|
173 glMatrixMode(GL_PROJECTION)
|
|
|
174 glLoadIdentity()
|
|
|
175 fovy <- 60.0
|
|
|
176 aspect <- width / height
|
|
|
177 znear <- 10
|
|
|
178 zfar <- 100
|
|
|
179 gluPerspective(fovy, aspect, znear, zfar)
|
|
|
180 }
|
|
|
181 @
|
|
|
182
|
|
|
183 Next, we need to setup the position of our camera in space using
|
|
|
184 the gluLookAt API which gets three vectors eye, center and up.
|
|
|
185
|
|
|
186 <<eval=FALSE>>=
|
|
|
187 setupCamera <- function(eye=c(0,0,-2),center=c(0,0,0),up=c(0,1,0))
|
|
|
188 {
|
|
|
189 glMatrixMode(GL_MODELVIEW)
|
|
|
190 glLoadIdentity()
|
|
|
191 gluLookAt(eye[[1]],eye[[2]],eye[[3]],center[[1]],center[[2]],center[[3]],up[[1]],up[[2]],up[[3]])
|
|
|
192 }
|
|
|
193 @
|
|
|
194
|
|
|
195 What follows is a short routine to draw a million 3D coordinates generated
|
|
|
196 by rnorm.
|
|
|
197
|
|
|
198 <<eval=FALSE>>=
|
|
|
199 draw <- function()
|
|
|
200 {
|
|
|
201 npoints <- 1000000
|
|
|
202 dims <- 3
|
|
|
203 data <- rnorm(npoints*dims)
|
|
|
204 type <- GL_DOUBLE
|
|
|
205 stride <- 0
|
|
|
206 glEnableClientState(GL_VERTEX_ARRAY_POINTER)
|
|
|
207 glVertexPointer(dims,type,stride,data)
|
|
|
208 stgrtindex <- 0
|
|
|
209 glDrawArrays(GL_POINTS, startindex, npoints)
|
|
|
210 glDisableClientState(GL_VERTEX_ARRAY_POINTER)
|
|
|
211 }
|
|
|
212 @
|
|
|
213
|
|
|
214 \section{Handling User-Input Events}
|
|
|
215
|
|
|
216 We will exchange the default behaviour of stopping after three seconds
|
|
|
217 with processing user-input events such as mouse, keyboard and joystick input
|
|
|
218 or pressing the close window button.
|
|
|
219
|
|
|
220 SDL uses a C structure to report user-interface events. One calls
|
|
|
221 a function called \verb@SDL_PollEvent()@ given the reference of this structure.
|
|
|
222
|
|
|
223 <<eval=FALSE>>=
|
|
|
224 processEvents <- function(env)
|
|
|
225 {
|
|
|
226 evt <- new.struct(SDL_Event)
|
|
|
227 quit <- FALSE
|
|
|
228 while(!quit) {
|
|
|
229 while( SDL_PollEvent(evt) ) {
|
|
|
230 type <- evt$type
|
|
|
231 if (type == SDL_QUIT) {
|
|
|
232 env$quit <- TRUE
|
|
|
233 } else if (type == SDL_MOUSEMOTION) {
|
|
|
234 motion <- evt$motion
|
|
|
235 cat(motion$xrel,",",motion$yrel,"\n")
|
|
|
236 }
|
|
|
237 }
|
|
|
238 }
|
|
|
239 }
|
|
|
240 @
|
|
|
241
|
|
|
242 \end{document}
|
|
|
243
|
|
|
244
|
