Conv nets

I knew that Convolutional neural networks has been succesfully applied in many comercial systems including ATM automatic check recognition. I also knew that Microsoft research guys works on it (prooflink) and seems to use it in MS Document Imaging OCR. But it was a very nice surprise for me that convnets are used by Google Street View to detect faces and license plates (prooflink).

And as a continue of this subject:

NEC has deployed ConvNet-based system in Japan for tracking customers in supermarket and recognizing their gender and age. Vidient Technologies has developed a ConvNet-based video surveillance system deployed in several airports in the US. France T´el´ecom has deployed ConvNetbased face detection systems for video-conference and other systems

(http://bit.ly/aYYesZ )

Here's another interesting application. Canon has developed VLSI chip for hardware implementation of Convnets.

P.S. If you have MS Office 2007 and want to try Document Imaging OCR for recognizing russian text you'll need to download CYRILLIC.SHP and copy it to Program Files\Common Files\Microsoft Shared\MODI\12.0.

File can be found here http://ifolder.ru/629971

Dropbox over proxy

If you're want to install dropbox on linux and you access internet through a proxy it's not a straightforward task. After downloading and installing application from dropbox main page it will try to download a proprietary daemon and will fail although you have set http_proxy environment variable. The solution found in the dropbox forums. You just need to manually download and unpack the latest but not stable version.

Google chart API

It was a pleasant surprise for me that google have an API to generate images with formulas using TeX formated strings. This is possible using Google Chart Tools.
You simply write link like this
"http://chart.apis.google.com/chart?cht=tx&chl=J^{*}_{t}=\left(%20j^{*}_{t}(1,1),j^{*}_{t}(1,2),j^{*}_{t}(1,3),%20\frac{\partial^2%20W_1}{\partial%20t^2},\frac{\partial^2%20W_2}{\partial%20t^2},\frac{\partial^2%20W_3}{\partial%20t^2}\right)^T,\\&j^{*}_{t}(1,k)=\left((P^T%20\ddot%20A_{k1}+%20\ddot%20A_{k2})P+2(2P^T%20\dot%20A_{k1}+%20\dot%20A_{k2})\dot%20P+\ddot%20A_{k3}%20\right)"
and get the following png picture:

CUDA Toolkit 3.0 now available

Good news!

There're some really delicious things in it:

• Multiple Copy Engine support
• Concurrent Kernel Execution
• C++ Class Inheritance and Template Inheritance support for increased programmer productivity
• CUDA Driver / Runtime Buffer Interoperability, which allows applications using the CUDA Driver API to also use libraries implemented using the CUDA C Runtime such as CUFFT and CUBLAS.
• CUBLAS now supports all BLAS1, 2, and 3 routines including those for single and double precision complex numbers
• New CUDA Memory Checker reports misalignment and out of bounds errors, available as a stand-alone utility and debugging mode within cuda-gdb
• On Linux, use cuda-gdb and cuda-memcheck, and check out the solutions from Allinea and TotalView that will be available soon.
• OpenCL Images support, for better/faster image filtering

C++ class inheritance is what I already was needed in my project. Higher performance and new debugging abilities also sounds great.

Resolving slow Eclipse problem

If you try to use Eclipse in newly installed Ubuntu, you might find that it's terribly slow. Just writing a dot after the variable can hang it for a minutes. The problem is in OpenJDK. You just need to install proprietary Sun Java to get a comfortable work. A very good howto about this could be found here.

New face database

Very nice surviliance cameras face database. 4160 images, different environment, different cameras, annotated positions of mouth, nose and eyes. And it's free! Though the request process is not simple, you'll have to fill an agreement and to be a full-time staff of the university.

CUDA Linux

Some Japanese company called Fixstars have released Yellow Dog Enterprise Linux for CUDA. I was amazed by this topic wandering what it could be in fact. I was imaging that Linux core computationally intensive operations accelerated by GPU etc. So what is it all about:

Key benefits of Yellow Dog Enterprise Linux for CUDA:

• YDEL for CUDA users can experience up to a 9% performance improvement in some applications.
• Comprehensive support is offered to paid subscriptions with our skilled team able to assist you with both Linux and CUDA.
• YDEL's unparalleled integrations means everything you need to write and run CUDA applications is included and configured.
• YDEL includes multiple versions of CUDA and can easily switch between them via a setting in a configuration file or an environment variable.
• Never worry about updates affecting your system, Fixstars offers YDEL users greater reliability with our strenuous test procedures that validate GPU computing functionality and performance.

So it's not about speeding up kernel, or something but rather Linux with convinient pack of
drivers, toolboxes, SDK's and addons helping developers to write and test CUDA programs.

3D animation

When I was a student I interested almost everithing about computers. And I always needed to try everything by myself. Among other computer connected areas there was a period of time when I was trying myself as a 3D modeler. I've tried 3DS, Lightwave, Poser, Bryce etc. Don't ask me where I've earned so much money to buy all that software, It was a dark times in Russia =). I used to make some simple movies, 3D objects, I've had some experience in OpenGL and Direct3D. But more than 6 years are pased since that time and now I needed something for making not very complicated 3D embedded in my software. I wasn't happy thinking about many days spent in remembering OpenGL and implementing all stuff. Then I tried to find some library helping me with this. Then my colleague adviced me to look at Panda project maintained by CMU. I was greatly surprised because it was very powerfull, crossplatform, very good documented, easy to use library. And it was free! Under BSD licence!!! It takes less than one day to implement all main functional I needed.

Take a look at their nice website

The linux version CudaCNN is awalible for download! Compiled and tested on Ubuntu 9.10 + CUDA 2.3 + Matlab 2009

There's a great plugin for Eclipse for CUDA development from Universitat Bayreuth. It can be found here. Thanks a lot.

I've released Convolutional Neural Network class of version 0.8 finaly. Since matlabcentral not accepts mex-files included into submissions, the most full version of it can be found here.

Among some bug fixes there're 3 most interesting features in this version.

First is GUI:

As can be seen there're RMSE (root mean squared error) and MCR (missclassification rate) plots. MCR is quite time expencive because it loops over some subset of training set to calucate, but as for me it worth it. MCR is much more informative than RMSE.

At the bottom of the window there're overall training and Hessian calculation progress bars. Very usefull thing as fom me. At the top right corner information about epoch, iteration, RMSE and MCR values. The last thing is Abort button. As opposed to previous version where you was able only to interrupt training by pressing "ctrl+c" loosing all training progress, abort button stops the training and saves all network parameters.

Second is new training modes. In the previous versions the Hessian digonal approximation (which is significantly speeds up covergence) has been calculated every iteration using so-called running estimate. But as stated by many authors Hessian is changes very slowly, so there's no need to recalculate it every iteration. So, I've added training mode at wich Hessian is calculated using some relatively small subset of training set and this repeated every few thousands of iterations. This trick really speeds up training without any sensible influence of covergence. Also I've added pure stochastic gradient as a third training mode.

Third and the main feature is CUDA-based CNN implementation. The CUDA technology has beecome very popular for speeding up computational intensive applications. And offcourse neural networks are seems to "been born" for this technology. So no surprise that neural network programmers tried to implement NNs even when there was no such convinient things as floating point support, or conditional execution in kernels. Implementation of classical fully-connected neural networks is quite trivial, because it's just a matrix by matrix multiplications. More interesting is implementations of convolutional neural networks, especially considering it's very exciting results in many applications such as handwriting digits recognition, face detection objects classification etc. Here what I've found on implementation of CNN's on GPU:

1. Kumar Chellapilla, Sidd Puri, Patrice Simard. High Performance Convolutional Neural Networks for Document Processing. 2006
2. Fabian Nasse, Christian Thurau, and Gernot A. Fink. Face Detection Using GPU-Based Convolutional Neural Networks. 2009
3. Dominik Scherer and Sven Behnke. Accelerating Large-scale Convolutional Neural Networks with Parallel Graphics Multiprocessors. In Proceeding of NIPS 2009 Workshop on Large-Scale Machine Learning: Parallelism and Massive Datasets, Whistler, Canada, December 2009
4. Narayanan Sundaramy, Anand Raghunathany, and Srimat T. Chakradhar. A framework for efficient and scalable execution of domain-specific templates on GPUs. 2009

In [1] colleagues from MS Research proposed a very interesting idea to represent a convolutions in C-layers as matrix multiplications. This requires some extra memory and calculations, but benefits are convinience, scalability and finally simulation and training speed. There was no CUDA in 2006 so they used shaders for CNN's speed up.

In [2] researchers describes very good practical application of CUDA for real-time face detection. Though training wasn't implemented on GPU.

In paper [3] scientists from Bonn University presented implementation of convolutional neural network using CUDA with both simulation and training. But from that paper seems that realization was hard-wired.

In [4] researchers from NEC presented software for various tasks on GPU and Convolutional neural networks is shown as example of such task. No information about training was provided there.

It should be noted that all these results presented only in papers so no code or even executables were given.

The only CUDA CNN's implementation known to me is this one. Good job, but it only implements simulation and is very hard-wired.

So all of that motivated me to write my own CUDA-based implementation of CNN's, which is scalable, trainable and publicaly awalible.

The main disadvantage in my realization is that it needs matlab as a source of CNN class object and a sink for results processing. But I'm going to make cudacnn as a separate library.