4

I found some papers and slides using deep learning for audio classification.

Some researches used spectrogram as the input of deep learning models.

I want to know exact and practical implementation.

And I found this slide.

Page 67

enter image description here

From my understanding, the node number of the first layer is 24 and the input is spectrogram of 24 different time periods.

For example, if a audio event is 2.4-second, the first node is spectrogram of 0~0.1 second, the second node is spectrogram of 0.1~0.2 second ...

Did I misunderstand ?

My question: if there is a 3.0-second audio event, how to classify it ?

yutseho
  • 1,639
  • 1
  • 15
  • 27
  • Possible duplicate of [How to train on and make a serialized feature vector for a Neural Network?](http://stackoverflow.com/questions/19419098/how-to-train-on-and-make-a-serialized-feature-vector-for-a-neural-network) – Nikolay Shmyrev Feb 03 '16 at 07:28

3 Answers3

3

I managed to classify time series data using Convolutional Neural Network. Convolution Neural Network is basically the same as Artificial Neural Network. The only difference is that, the input to the ANN must be convolved first to extract specific features. In an intuitive way, convolution operation basically highlights specific features of some data. It is best depicted by flashlight shined through different parts of the images. By doing so, we can highlight specific features of the image.

That's the main idea of CNN. It is inherently designed to extract spatial features. The convolution operation is usually stacked, which means you have (row,column,dimensions) so the output of the convolution is 3 dimension. The downside of this process is large computation time. To reduce that, we need pooling or downsampling which basically reduce the size of the feature detectors without losing essential features/information. For example before pooling you have 12 of 6,6 matrix as feature detectors. And after pooling you have 12 convolved data with size of 3,3. You can do these two steps over and over again before flattening which basically squash all those into (n,1) dimensional array. Afterwards, you can do normal ANN steps.

In short, the steps to classify time series data can be done using CNN. Here are the steps:

1.Convolution
2.Pooling
3.Flattening
4.Full connection (normal ANN steps)

You can add convolution and pooling layers as much as you like, but watch out for training time. There's this video by my favourite youtuber, Siraj Raval. By the way, I suggest you to use Keras for Deep Learning. Hands down the easiest deep learning library to use. Hope it helps.

Amajid Sinar
  • 166
  • 2
  • 9
1

You should use Kaldi. CTC takes care of temporal resolutions.

Morteza Shahriari Nia
  • 1,392
  • 18
  • 24
0

I trained a CNN to detect the language that was spoken in an audio recording. It currently supports 176 languages with an accuracy of 98.8%. I have a fairly well commented Jupyter Notebook on my GitHub account: Spoken Language Classifier.

I expect that this is what you're looking for. Some of the things I learned include:

  • The architecture doesn't need to be recurrent, because time can be encoded on the x axis. For a non-recurrent CNN the length of your input has to be fixed though.

  • Spectrograms are semantically different to photographs in many respects. Popular architecures that work well for photos, may be complete overkill for spectrograms.

  • Experiment with different resolutions for x and y individually. My first assumption that the time axis needs higher resolution than the frequency axis was wrong in my use case.

  • Use a mel-spectrogram to give higher resolution to lower frequency. Our hearing works exponentially and not linear.

pietz
  • 2,093
  • 1
  • 21
  • 23