I have been developing a graphical user interface (GUI) in MATLAB to present experimental stimuli to participants and record their responses in a variety of formats. MATLAB makes it pretty easy to collect self-report data using the uicontrol function, which can be configured as an edit box, drop-down box, check box, list box, button, or slider. However, I also wanted to collect behavioral data using the computer’s built-in webcam and microphone. As it turns out, recording synchronized audio and video data in MATLAB is surprisingly difficult. This post will describe the solution I found and provide source code that you can build upon.

###### Failed Attempts

The first thing I tried was to use the videoinput function as described in this post by Richard Barnes. I was pretty hopeful that this would work and, after installing the OS Generic Video Interface support package, I did get MATLAB to find the webcam built into my laptop and record images both to disk (i.e., a video file) and to memory (i.e., a MATLAB variable). The source code for this attempt is listed below. However, I was never able to get a stable frame rate using this method and it didn’t incorporate synchronized audio recording from my microphone.

% See what image acquisition devices are available
>> devices = imaqhwinfo
MATLABVersion: '9.0 (R2016a)'
ToolboxName: 'Image Acquisition Toolbox'
ToolboxVersion: '5.0 (R2016a)'

% See default format for the device named winvideo
>> info = imaqhwinfo('winvideo',1)
DefaultFormat: 'MJPG_1280x720'
DeviceFileSupported: 0
DeviceName: 'USB2.0 HD UVC WebCam'
DeviceID: 1
VideoInputConstructor: 'videoinput('winvideo', 1)'
VideoDeviceConstructor: 'imaq.VideoDevice('winvideo', 1)'
SupportedFormats: {1x12 cell}

% Create video input and preview it
inp = videoinput('winvideo',1,'MJPG_1280x720');
preview(inp);

% Configure video input to collect 5 sec of video
src = getselectedsource(inp);
frt = str2double(src.FrameRate);
inp.FramesPerTrigger = round(frt * 5);
inp.TriggerFrameDelay = 0;

% Configure video input to write to video file
file = VideoWriter('testvid.avi');
inp.LoggingMode = 'disk';
inp.DiskLogger = file;

% Start the recording
start(inp);
wait(inp,Inf);

% Clean up after recording
delete(inp);
clear inp;


###### Working Solution

The solution I ended up most satisfied with was to outsource the problem and use the ffmpeg program to record the audiovisual data. Because ffmpeg is called from the command line, it is very easy to control it from MATLAB using the dos or system functions. I downloaded a Windows build of ffmpeg and extracted the ffmpeg.exe file to the MATLAB working directory. I then had to figure out how to record from webcam to a file using ffmpeg. Luckily, I found this helpful page about the DirectShow API and some decent ffmpeg documentation. Below, I provide the code that ended up suiting my needs best. I chose the highest resolution and framerate that my camera could produce (i.e., 1280×720 and 30 fps) and set the audio buffer size to 80 ms as recommended by this article. I chose to record for only 10 seconds (i.e., -t 10) as a quick, initial test. I also chose to compress the recorded video using libx264, the veryfast preset, and a constant rate factor of 25 (i.e., -crf 25) to yield small but good quality files. Any of these options could be changed for different applications, or parameterized using the sprintf function.

% List the DirectShow devices attached to this computer (output trimmed)
>> dos('ffmpeg -list_devices true -f dshow -i dummy')
[dshow @ 00000000009724a0] DirectShow video devices
[dshow @ 00000000009724a0]  "USB2.0 HD UVC WebCam"
[dshow @ 00000000009724a0] DirectShow audio devices
[dshow @ 00000000009724a0]  "Microphone (Realtek High Definition Audio)"

% Show options for the desired video capture device (output trimmed)
>> dos('ffmpeg -f dshow -list_options true -i video="USB2.0 HD UVC WebCam')
[dshow @ 0000000000a524a0] DirectShow video device options (from video devices)
[dshow @ 0000000000a524a0]   vcodec=mjpeg  min s=640x480 fps=15 max s=640x480 fps=30
[dshow @ 0000000000a524a0]   vcodec=mjpeg  min s=160x120 fps=15 max s=160x120 fps=30
[dshow @ 0000000000a524a0]   vcodec=mjpeg  min s=176x144 fps=15 max s=176x144 fps=30
[dshow @ 0000000000a524a0]   vcodec=mjpeg  min s=320x240 fps=15 max s=320x240 fps=30
[dshow @ 0000000000a524a0]   vcodec=mjpeg  min s=352x288 fps=15 max s=352x288 fps=30
[dshow @ 0000000000a524a0]   vcodec=mjpeg  min s=1280x720 fps=15 max s=1280x720 fps=30

% Record 10 s of video and audio to the specified file with the requested parameters
dos('ffmpeg -f dshow -video_size 1280x720 -framerate 30 -audio_buffer_size 80 -i video="USB2.0 HD UVC WebCam":audio="Microphone (Realtek High Definition Audio)" -t 10 -c:v libx264 -preset veryfast -crf 25 -vf "fps=30" test_vid2.mp4');