# How can I pair a scrolling spectrogram with high resolution video?

I’m looking for a way to review very high resolution video alongside a scrolling spectrogram of the associated audio track. Some DAWs that have spectrograms (Adobe Audition,Reaper) allow for simultaneous video but the video won’t playback at full frame rate or resolution. VLC can handle the video and has audio visualizations, but not a spectrogram. Anyone know of a tool?

It isn’t enough to route the audio to a separate program running a real time display. I need to be able to scroll around the video and see the spectrogram as I do.

As an alternative, I’ve been making a movie of a spectrogram using screen recording, then overlaying this on the video in Adobe Premiere. This actually works very well for playback but it’s a pain to make the movies. I just have to screen record them and manually sync them up. Surely someone has a tool that can generate a video of a scrolling spectrogram without screen recording and faster than real-time. Thoughts?

• For anyone else who's interested, Boris [link] (boris.unito.it) looks like a very user-friendly way to do this and it seems it would handle the high-res video smoothly. Sadly they don't support Mac anymore so I'm unable to make it work. Linux or Windows, it might be the answer. Aug 18 at 20:21
• I edited the title of your question a bit to make it more detailed/specific and better fit the SE Q/A format. I hope to have maintained the essence of your question, but if not, please comment here or edit back. For more info see: How do I write a good title? Aug 25 at 16:41
• sometimes comments can be deleted over time or get hidden, your suggestion would make a great answer for others looking to solve the same problem. You are welcome to answer your own question on SE, so would you mind turning your above comment about Boris into an answer? Aug 25 at 16:44
• Happy to do so but how do I change the comment into an answer? Do I repost it as an answer? Aug 26 at 17:34
• yup exactly...You can just scroll down to the bottom and add it as an answer (there should be a white answer box you can type in or a blue button that says "answer your own question" or something like that. Then if folks try it and it helps them, they can vote for it! Aug 26 at 17:52

I asked a question in the similar topic couples of weeks ago: Annotate acoustic and visual events of a multimodal signal.

To get real good quality scrolling spectrogram in video I use dynaSpec package in R that allows you to generate MP4 video of a spectrogram: https://marce10.github.io/dynaSpec/.

When I do a video recording of a vocalizing animal and I want to review both the visual behaviour and the spectrogram of the vocalization, I first create a scrolling spectrogram using dynaSpec. Then, I combine the video recording with the scrolling spectrogram using Davinci Resolve software (free version).

• This is a great answer to the second option, which is creating a scrolling video and adding it with video editing software. I'd prefer some software that just loaded the video and played a spectrogram in real-time. But your solution to the creation of scrolling spectrograms is a definite improvement over the way I create spectrogram movies. Right now I'm stuck screen recording. Now all I have to do is learn R. Aug 15 at 15:39
• I have a collaborator trying to make this work but he's running into difficulties. I'll report back if he gets it up and running. Meanwhile, I'm just screen recording Raven windows so I can move forward. That may end up being the easiest solution that is accessible to me after all. Aug 18 at 20:23
• My collaborator got the R package working, but it is painfully slow. He did a three-second file and it took 3 minutes. We set up an hour-long file and let it go overnight. It says it still has more than 7 days. Does this jive with anyone else's experience with this package? Or is something wrong on our side? Aug 19 at 15:23
• Hi @MarcD , I haven't used this package to date, but R can be notoriously slow with some (seemingly) straightforward processes. I'm assuming you're using a high performance computer to implement the package? Aug 22 at 21:46
• Hello, it is right that R is very slow to create such MP4 file. I think the package was created just for making a short video of scrolling spectrogram. Even with my gaming computer that is very powerfull, it takes ages to create scrolling spectro, while creating long videos with Davinci Resolve as example, is much faster. Aug 23 at 9:42

There is a similar question on this Video SE. One suggestion/answer uses ffplay (linux?) to combine video with scrolling spectrogram.

• Thank you. I wish I could try that but I think this method is technically out of reach for me. No Linux knowledge. Aug 15 at 15:41

I’m not sure that this will fully solve your problem, but for the alternative you have listed (generate scrolling spectrogram + overlay w/ video in adobe premiere), ffmpeg is a useful tool to consider. The “showspectrum” function from ffmpeg is a great option for producing scrolling spectrogram videos from a .wav by running a (relatively) simple shell script. This tool gives you lots of flexibility with spectrogram parameters, aesthetics, etc. (a good example is found here). You can create a similar .sh script for your needs and run from the command line to generate your scrolling spectrogram video (or many such videos) without screen recording.

This of course would still require syncing with video, but will at least automate and greatly speed up your generation of scrolling spectrogram videos. As another commenter suggested, dynaSpec (R package) also appears to be a useful tool for this purpose but I do not personally have experience working with that package.

• Thank you. It's not clear to me how much time that would save me vs. screen recording, especially as I'd need to learn ffmpeg first. I think I'm just going to record the Raven window. It is realtime, but with my long recordings, I can just let it run. The screen recorder I have can just stop after a set period of time, so I don't have to stick around. I may look back at that option when I have more data, but for a few days worth of recording, I'll probably just go with what I know. Aug 18 at 20:15

I do not know how adaptable this set of tools is beyond the specific CATS animal-borne tagging application, but this paper provides a set of code/tools that pairs video, acoustic, and accelerometer data:

https://animalbiotelemetry.biomedcentral.com/track/pdf/10.1186/s40317-021-00256-w.pdf

The paper points to the GitHub repositories for the code and a workshop with tutorials and wikis to guide new users.

• Wow! That looks powerful and interesting. It could be overkill and would be a steep curve for me. Maybe for the next phase of my project where I hope to have more data and some little more technical support who could pull it together. Thanks. Aug 18 at 20:16

If you end up working with any of the tools in the GitHub repository mentioned by Selene and run into any problems/questions, let me know and we can work through them together!

• Thanks so much for the offer. It looks like an amazing tool. I wonder if it is overkill given that I only have video and spectrograms. Also, if it would handle the 5.7 k video. Anyway, I think it is too much for me to pull off at this stage. It's a preliminary study with limited hours of data. I think I'm going to bite the bullet and screen record. Maybe in the next phase with more data. Aug 18 at 20:18