Category Archives: Sphinx4

A look on Sphinx3's initialization

I worked on Sphinx 3 a lot.  In these days, it was generally regarded as an "old-style" recognizer as compared to Sphinx 4 and PocketSphinx.   It is also not support officially by the SF's guys.

Coders of speech recognition think a little bit different.  They usually stick to a certain codebase which they feel comfortable with.   For me, it is not just a personal preference, it also reflects how much I know about a certain recognizer.  For example, I know quite a bit of how Sphinx 3 performs.   In these days, I tried to learn how Sphinx 4 fare as well.   So far, if you ask me to choose an accurate recognizer, I will still probably choose Sphinx 3, not because the search technology is better (Sphinx 4 is way superior), but because it can easily made to support several advanced modeling types.  This seems to be how the 2010 developer meeting concluded as well.

But that was just me. In fact, I am bullish on all Sphinx recognizers.  One thing I want to note is the power of Sphinx 4 in development.  There are many projects are based on Sphinx 4.  In these days, if you want to get a job on speech recognizer, knowing Sphinx 4 is probably a good ticket.  That's why I am quite keen on learning it more so hopefully I can write on both recognizers more.

In any case, this is a Sphinx 3's article.  I will probably write more on each components.   Feel free to comments.

How Sphinx3 is initialized:

Here is a listing of function used on how Sphinx 3 is initialized I got from Sphinx 3.0.8.  Essentially, there are 3 layers of initialization, kb_init, kbcore_init and s3_am_init.  Separating kb_init and kbcore_init probably starts very early in Sphinx 3.  Whereas separating s3_am_init from kbcore_init was probably from me. (So all blames on me.)  That is to support -hmmdir.

 kb_init  
-> kbcore_init (*)
-> beam_init
-> pl_init
-> fe_init
-> feat_array
-> stat_init
-> adapt_am_init
-> set operation mode
-> srch_init
kbcore_init
-> Look for feat.params very early on.
-> logmath_init
-> feat_init
-> s3_am_init (*)
-> cmn_init
-> dict_init
-> misc. models init
mgau_init such as
-> subvq_init
-> gs_read
-> lmset_init
-> fillpen_init
-> dict2pid_build <- Should put into search
s3_am_init
-> read_lda
-> read in mdef.
-> depends on -senmgau type
.cont. mgau_init
.s2semi. s2_semi_mgau_init
if (-kdtree)
s2_semi_mgau_load_kdtree
.semi or .s3cont.
ms_mgau_init
-> tmat_init
Note:
  • -hmmdir override all other sub-parameters. 

Arthur

January 2013 Write-up

Miraculously, I still have some momentum for this blog and I have kept on the daily posting schedule.

Here is a write up for this month:  Feel free to look at this post on how I plan to write this blog:

Some Vision of the Grand Janitor's Blog

Sphinx' Tutorials and Commentaries

SphinxTrain1.07's bw:

Commentary on SphinxTrain1.07's bw (Part I)
Commentary on SphinxTrain1.07's bw (Part II)

Part I describes the high-level layout, Part II and describe half the state network was built.

Others:
Acoustic Score and Its Sign
Subword Units and their Occasionally Non-Trivial Meanings

Sphinx4:
Sphinx 4 from a C background : Material for Learning

News

Goldman Sachs not Liable
Aaron Swartz......

Other writings:

On Kurzweil : a perspective of an ASR practitioner

Enjoy!

Arthur

Where to start when tracing source code of a speech recognition toolkit?

Modern speech recognition software are complicated piece of software.  To understand it, you need to have some basic understanding of the principle of speech recognition, as well as some ideas on the programming language being used.

By now, you may hear a lot of people say they know about a speech recognizer.   And by now, you probably realize that most of these people have absolutely no ideas what's going on inside a recognizer.   So if you are reading this blog message, you are probably telling yourself, "I might want to trace the codebase of some recognizers' code." Be it Sphinx, HTK, Julius, Kaldi or whatever codebase you are looking at.

For the above toolkits, I will say I only know in detail about Sphinx,  probably a little bit about HTK's HVite.  But I won't say the same for others.  In fact, even in Sphinx, I only know intimately about Sphinx 3/SphinxTrain/sphinxbase triplet.   So just like you, I hope to learn more.

So here it begs the question: how would you trace a speech recognition toolkit codebase? If you think it is easy, probably because you worked in speech recognition for a while and you probably shouldn't read this post.

Let's just use sphinx as an example, there are hundreds of files in each component of Sphinx.   So where should you start?    A blunt approach would be reading each of the file one by one.   That's not a smart the way.   So here is a suggestion for you : focus on the following four things,

  1. Viterbi algorithm
  2. Workflow of training
  3. Baum-Welch algorithm. 
  4. Estimation algorithms of language models. 
When you know where the Viterbi algorithm is, you will soon figure out how the feature vector is generated.  On the same vein: if you know where the Baum-Welch algorithm, you will probably know how the statistics are generated.   If you know the workflow of the training, then you will understand the how the model is "evolved".   If you know how the language model is estimated, then you would have understanding of one of the most important heuristic of the search. 
Some of you may protest, how about the front-end? Isn't that important too?  True, but not when you try to understand a codebase.  For all practical purpose, a feature vector is just an N-dimensional vector.  The waveform is just an NxT matrix.   You can certainly do a lot of fancy things on this NxT matrix.   But when you think of Viterbi and Baum-Welch, they probably just read the frames and then calculate Gaussian distribution.  That's pretty much it's how much you want to know a front-end. 
How about adaptation algorithms?  That I think it's important.   But it should probably go after understanding of the major things in the code.   Because no matter whether you are doing adaptation online or doing this in speaker adaptive training.  It is something on top of the Baum-Welch algorithm.   Some implementation stick adaptation within the Baum-Welch executable.  There is certainly nothing wrong about it.   But it is still a kind of add-on. 
How about decoding API?  Those are useful things to know but it is more important when you just need to write an application.  For example, in Sphinx4, you just need to know how to call the Recognizer class.  In sphinx3, live_decode is what you need to know.   But only understanding those won't give you too much insights of how the decoder really works. 
How about the data structure?  Those are sort of important and should be understood when you try to understand a certain algorithm.   In the case of languages such as Java and C++, you should probably take notes on a custom-made data structure.  Or whether the designer call a specific data structure libraries.  Like Boost in C++. 
I guess this pretty much sums it all.  Now let me get back to one non-trivial item on the list, which is the workflow of training.   Many of you might think that recognition systems differ from each other because they have different decoders.  Dead wrong!  As I stressed from time to time, they differ because they have different acoustic models and language models.  So that's why in many research labs, much effort was put on preserving the parameters and procedures of how models is trained.  Much effort was also put to fine tuned this procedure.  
On this part,  I got to say open source speech recognition still has long long way to go.  For starter, there is no much sharing of recipes among speech hobbyists.   What many try to do is to search for a good model.   If you don't know how to train a model, you probably don't even know how to improve it for you own project.   
Arthur

Sphinx 4 from a C Background : Setting up Eclipse as the IDE

This is another baby step on how one can learn about Sphinx 4.   As I mentioned in the previous post,  it is nicer to use an IDE when you use Java code.  Since I have some exposure in Eclipse, I choose it as an example on how to setup a Sphinx 4 build.

Before I go on there were many posts, written by others, discuss the procedure.  You may take a look of them as well.

You will also need to know how to install JSAPI (link).  It is crucial to get the compilation right. 

Eclipse as a Development Environment

If you never use Eclipse before, it is a little bit like a more versatile version of Emacs.   It's major use is on Java but lately there are more and more people use it as IDE for C/C++ as well.  Not to say there are more different development packages for different programming languages. 
If you come from background such as emacs/vi development, one thing you need to know is that shortcuts are quite different from your current platform.  That takes some time to adapt to but generally I think the advantage worth the cost.
Another thing you might want to be mentally prepare, Eclipse's Java compilation doesn't generate build log.  Instead it will generate a list of errors in compilation.   They are basically equivalent thing.  Though, if you are used to Visual C++ type of IDE with an error log, you won't get what you want.  
To me, those are minor nuisances, using Eclipse to browse code has the extra advantage of readily-made documentation as well as a flatten structure.  Those features will save you many keystrokes if compared to using vanilla emacs. 
In my description, I am using Eclipse Juno.  Hopefully it won't change too much by the time you are compiling the code.  Of course, if there is popular demand, I might write another post which describe later version of Eclipse as well.

The compilation in High Level

Building Sphinx 4 essentially means the following four tasks:
  1. Downloading Sphinx4 source code
  2. Install JSAPI.
  3. Incorporate the proper libraries. 
  4. Do the build. 
In my case, I slightly stumbled on 1, naturally, just like you, I was thinking "well, why JSAPI something separate from the codebase?"  Of course, if you worked in Java before, there are many projects required you to build with external codebase.  So I don't think too bad. 
So let me go through the procedure of the build. 
Downloading Sphinx4 source code from Subclipse
  • A plain simple svn command is fine, downloading the tarball will give you a more stable version.  I will suggest a more attractive option is to use SVN module of Eclipse, subclipse.   To do that, you may want to follow "Downloading Subclipse" from Setting up Development Environment .   (Notice that there was a typo in the post should be "tigris" instead "trigris" on the location field.) 
  • Once you finished checking out Subclipse.  Start a new Project 
    • New -> Project -> SVN -> Checkout Projects from SVN
  • Choose "Create a New Repository Location"
  • Remember to only download trunk/sphinx4 (Note: there are many branches and location, for starter, you will be interested how the trunk look like.)
Once you check out the code, in your Package Explorer (Alt-Shift-Q -> P) will look like this. 

Package Explorer View after code is check out from SVN
Now you might notice that there is a red question mark besides the sphinx4 project (I named it "sphinx4_grandjanitor" but you can name it whatever you want.) You might also notice that in your Problem screen, there are 2 errors :
Now this is really because lib/jsapi.jar wasn't installed correctly.  So the next step is to install jsapi.jar

Install JSAPI

I tried the install of both Windows Vista and Linux.  In windows, go to sphinx4lib and type

> jsapi.exe

Then accept the license.

In Linux, in the same directory.  do

> sh jsapi.sh

One common problem for Linux here: you need to install uudecode if you want to install jsapi.  In that case, try to install sharutil.  On Ubuntu, it works for me when I do

> apt-get install sharutil 

At this point you should see your directory should have a file named jsapi.jar

Incorporate the proper libraries

This is another part which took me a while.  Before you go on to configure your path, you need to do one more step to make to configure libraries.   In Eclipse, right click you Sphinx4/lib directory and choose Refresh first.  This will make jsapi.jar appears your Package Explorer.  It should look like this:

When JSAPI.jar is properly installed

Then, you can change the build path, go to your project again, right click and choose Build Path -> Configure Build, Libraries, choose Add Jar, then add the libraries you need.

Now.... wait, what are the jar files we need again?

Yeah, so this is another place which can cause confusions.  In fact, because Sphinx has expanded its code from time to time, so the answer of which jar files to add depends.   As of Dec 28, 2012, you should add

  • junit
  • jsapi
  • js
  • fst

This list will likely to grow in future.  I am also pretty sure you might need to do different things if you want to compile in a different setting or write your own code.

Do the build

In modern Eclipse, building should be automatic, what you should see should be 0 errors but many warnings.    I generally don't approve of warnings but as a developer, it's pretty tough to eliminate them all.

Conclusion

There you have it, a little guide on Sphinx 4 compilation with Eclipse.  Notice that this guide may or may not fit your purpose because I focus on downloading the code from Subclipse.   Doing a Link Source should do the trick if you want to incorporate the code yourself.   I might do another post later but the web has many articles described this already, you should be able to find a set of good instructions. 
Arthur
Related Posts: 




Me and CMU Sphinx

As I update this blog more frequently, I noticed more and more people are directed to here.   Naturally,  there are many questions about some work in my past.   For example, "Are you still answering questions in CMUSphinx forum?"  and generally requests to have certain tutorial.  So I guess it is time to clarify my current position and what I plan to do in future.

Yes, I am planning to work on Sphinx again but no, I probably don't hope to be a maintainer-at-large any more.   Nick proves himself to be the most awesome maintainer in our history.   Through his stewardship, Sphinx prospered in the last couple of years.  That's what I hope and that's what we all hope.    
So for that reason, you probably won't see me much in the forum, answering questions.  Rather I will spend most of my time to implement, to experiment and to get some work done. 
There are many things ought to be done in Sphinx.  Here are my top 5 list:
  1. Sphinx 4 maintenance and refactoring
  2. PocketSphinx's maintenance
  3. An HTKbook-like documentation : i.e. Hieroglyphs. 
  4. Regression tests on all tools in SphinxTrain.
  5. In general, modernization of Sphinx software, such as using WFST-based approach.
This is not a small undertaking so I am planning to spend a lot of time to relearn the software.  Yes, you hear it right.  Learning the software.  In general, I found myself very ignorant in a lot of software details of Sphinx at 2012.   There are many changes.  The parts I really catch up are probably sphinxbase, sphinx3 and SphinxTrain.   One PocketSphinx and Sphinx4, I need to learn a lot. 
That is why in this blog, you will see a lot of posts about my status of learning a certain speech recognition software.   Some could be minute details.   I share them because people can figure out a lot by going through my status.   From time to time, I will also pull these posts together and form a tutorial post. 
Before I leave, let me digress and talk about this blog a little bit: other than posts on speech recognition, I will also post a lot of things about programming, languages and other technology-related stuffs.  Part of it is that I am interested in many things.  The other part is I feel working on speech recognition actually requires one to understand a lot of programming and languages.   This might also attract a wider audience in future. 
In any case,  I hope I can keep on.  And hope you enjoy my articles!
Arthur

Sphinx4 from a C background : Installation of Eclipse

That's another baby step but I guess Eclipse installation is much less painful these days.

When I used Eclipse back in 2008, it was rather difficult to download and install.   Part of the reason is that the software house I worked with didn't have a strong culture of documentation.

Downloading Eclipse Juno for Java Developer was pretty easy.  My next step is to incorporate Sphinx 4 directory and do a compilation.

Arthur

Sphinx4 from a C background : first few steps

As I set out earlier,  one of my goals is to grok all of the components.  I challenged myself to work with Java, which I feel less proficient than my C/C++/Python/Perl.

What should you think when you go from one language to another?  One and only one answer : don't make a judgement too early.  
For example, compilation of Sphinx4 takes 4 steps:
  1. Download and install JDE. 
  2. Download and install ant. 
  3. run ant
If you haven't used JDE, ant or never look at a build.xml, you would feel a bit overwhelmed.    But be patient, there are a lot of goodies of Java.  Most of them are very well thought in terms of software engineering. 
I followed the process.  Woa,  Sphinx 4 is now at beta 6 and it grows to 366 files.   Sounds like groking it will take some time then. 
So what would be your strategy if you want to go forward to understand a Java project such as Sphinx4?   My suggestion: download a good IDE such as Eclipse or NetBeans.
If you are like me, coming from a emacs background, learning Eclipse would take you sometime as well.   But again: don't make a judgement too early.  Eclipse is nice in its own way.  (At least it's not Visual X.....)    
Practically, using Eclipse to understand the code also has its advantage.  Unlike C-package organization, Java software usually has deep directory hierarchy.  Using emacs would definitely cause you more keystrokes.  The only exception I know of is JDEE.  That again will take you some setup time.
In any case, I got it started.  So, my next goal is to go through all materials of Sphinx 4 again.  This time I demand myself to grok.   I will start from the Sphinx 4 documentation page.  Then expand to source code-level of undersand. 
Arthur