A General Artificial Neural Network Extension for HTK

Similar documents
Machine Learning: Chenhao Tan University of Colorado Boulder LECTURE 8

Regularized Linear Models in Stacked Generalization

SPEED AND TORQUE CONTROL OF AN INDUCTION MOTOR WITH ANN BASED DTC

Artificial-Intelligence-Based Electrical Machines and Drives

From Developing Credit Risk Models Using SAS Enterprise Miner and SAS/STAT. Full book available for purchase here.

Assignment 3 solutions

Text Generation and Neural Style Transfer

Calibration. DOE & Statistical Modeling

Predicting Solutions to the Optimal Power Flow Problem

Lecture 7. LVCSR Training and Decoding (Part A) Bhuvana Ramabhadran, Michael Picheny, Stanley F. Chen

Intelligent Fault Analysis in Electrical Power Grids

Data Mining Approach for Quality Prediction and Improvement of Injection Molding Process

Introduction to Abaqus/CAE. Abaqus 2018

Probabilistic Modeling of Fatigue Damage in Steel Box-Girder Bridges Subject to Stochastic Vehicle Loads

CSE 40171: Artificial Intelligence. Artificial Neural Networks: Neural Network Architectures

QNX Automotive Overview Senthil Kumar, Application Engineer

Statistical Learning Examples

Permanent Magnet Motors for ESP Applications Updating the Track Record of Performance. Lorne Simmons VP Sales & Marketing

Inverter control of low speed Linear Induction Motors

The State of Charge Estimation of Power Lithium Battery Based on RBF Neural Network Optimized by Particle Swarm Optimization

Incremental Joint Extraction of Entity Mentions and Relations

Intelligent CAD system for the Hydraulic Manifold Blocks

Available online at ScienceDirect. Energy Procedia 36 (2013 )

The Session.. Rosaria Silipo Phil Winters KNIME KNIME.com AG. All Right Reserved.

Robust Fault Diagnosis in Electric Drives Using Machine Learning

Face Detection and Bi-Modal Authentication

Multi Core Processing in VisionLab

White paper: Pneumatics or electrics important criteria when choosing technology

WeDo 2.0. Science & Technologies. Effectively implement the. Freecall:

P. D. Belapurkar, S.D. Mohite, M.V. Gangawane, D. D. Doltode (Department of Mechanical, M.E.S. College of Engineering, S.P. Pune University, India)

Early Stabilizing Feature Importance for TensorFlow Deep Neural Networks

Contents INTRODUCTION...

Autonomous inverted helicopter flight via reinforcement learning

QCF level: 4 Credit value: 15

FUEL DELIVERY IGNITION ANGLE CONTROL BOOST CONTROL TECH INFO

Composites Modeler for Abaqus/CAE. Abaqus 2018

MAX PLATFORM FOR AUTONOMOUS BEHAVIORS

Assembly Line Balancing using Artificial Neural Network: A Case Study of Tricycle Assembly Line

G. K. VENAYAGAMOORTHY

Full Vehicle Simulation for Electrification and Automated Driving Applications

K. Shiokawa & R. Takagi Department of Electrical Engineering, Kogakuin University, Japan. Abstract

NTRU Challenge - Answers

Using Opal-RT Real-Time Simulation and HIL System in Power and Energy Systems Research

Building Fast and Accurate Powertrain Models for System and Control Development

REAL TIME TRACTION POWER SYSTEM SIMULATOR

Vehicle's Velocity Time Series Prediction Using Neural Network

What s new. Bernd Wiswedel KNIME.com AG. All Rights Reserved.

Multiobjective capacitated arc routing problem

EJ2410 HYBRID VEHICLE DRIVES

index changing a variable s value, Chime My Block, clearing the screen. See Display block CoastBack program, 54 44

Electronic Load Sensing for Tractors

Experimental Verification of the Implementation of Bend-Twist Coupling in a Wind Turbine Blade

Performance Simulation of Energy Storage Technologies for Renewable Energy Integration

Statistical Estimation Model for Product Quality of Petroleum

Embedded system design for a multi variable input operations

SOC estimation of LiFePO 4 Li-ion battery using BP Neural Network

OFFSHORE Diesel Fuel Treatment Technical Data By:

DEPARTMENT OF COMPUTER SCIENCE AND ELECTRICAL ENGINEERING GRADUATE SCHOOL OF SCIENCE AND TECHNOLOGY KUMAMOTO UNIVERSITY KUMAMOTO JAPAN

Statistical and Artificial Neural Network based Analysis of Faults in an Automobile Engine

Comparison of Karanja, Mahua and Polanga Biodiesel Production through Response Surface Methodology

CONCEPTUAL CAR DESIGN AT BMW WITH FOCUS ON NVH PERFORMANCE

ValveLink SNAP-ON Application

Manufacturing Quality Control of High-Pressure Composite Cylinders by Acoustic Emission

Arcing prevention by dry clean optimization at Shallow Trench Isolation (STI) Etch in AMAT MxP by use of plasma parameters

Scheduling. Purpose of scheduling. Scheduling. Scheduling. Concurrent & Distributed Systems Purpose of scheduling.

CMPEN 411 VLSI Digital Circuits Spring Lecture 20: Multiplier Design

EMC System Engineering of the Hybrid Vehicle Electric Motor and Battery Pack

Automated Driving: Design and Verify Perception Systems

PROCEDURES FOR ESTIMATING THE TOTAL LOAD EXPERIENCE OF A HIGHWAY AS CONTRIBUTED BY CARGO VEHICLES

Optimization of Chromatogram Alignment Using A Class Separability Criterion

What s Cooking. Bernd Wiswedel KNIME KNIME.com AG. All Rights Reserved.

A simulator for the control network of smart grid architectures

HYBRID ELECTRIC VEHICLE SYSTEM MODELING AND CONTROL

Data envelopment analysis with missing values: an approach using neural network

Analysis. Techniques for. Racecar Data. Acquisition, Second Edition. By Jorge Segers INTERNATIONAL, Warrendale, Pennsylvania, USA

Components and tooling to reduce complexity and cost in E/E powertrain system design for Hybrid electric Vehicles

Up scaling Agent Based Discrete Choice Transportation Models using Artificial Neural Networks

Lithium Ion Battery Charging Using Bipolar Transistors

SPECIFIC TECHNICAL CRITERIA

CMPEN 411 VLSI Digital Circuits Spring Lecture 22: Memery, ROM

Energetic Macroscopic Representation and Energy Management Strategy of a Hybrid Electric Locomotive

An Autonomous Braking System of Cars Using Artificial Neural Network

ANALYTICAL EVALUATION OF ENGINE AND VEHICLE HARDWARE EFFECTS ON VEHICLE RESPONSE. Drew Raftopoulos

Accelerated Testing of Advanced Battery Technologies in PHEV Applications

Symmetric Key Broadcast Encryption: State-of-the-Art

Evaluation Certificate

Unidrive M600 High performance drive for induction and sensorless permanent magnet motors

Industrial furnaces. Innovative solutions for your success

Enhancing Energy Efficiency of Database Applications Using SSDs

Indian Journal of Science and Technology Vol. 5 No.3 (Mar 2012) ISSN:

Analysis and Design of the Super Capacitor Monitoring System of Hybrid Electric Vehicles

Adaptive Fault-Tolerant Control for Smart Grid Applications

D ISM Lubricant Performance Test. Report Packet Version No. Method. Conducted For:

Design and evaluate vehicle architectures to reach the best trade-off between performance, range and comfort. Unrestricted.

IJSRD - International Journal for Scientific Research & Development Vol. 4, Issue 06, 2016 ISSN (online):

PSO project EaseWind Enhanced ancillary services from Wind Power Plants. Anca D. Hansen DTU Wind Energy

International Journal of Advance Research in Engineering, Science & Technology

The design and implementation of a simulation platform for the running of high-speed trains based on High Level Architecture

Speed Control of BLDC motor using ANFIS over conventional Fuzzy logic techniques

Transcription:

A General Artificial Neural Network Extension for HTK Chao Zhang & Phil Woodland University of Cambridge 15 April 2015

Overview Design Principles Implementation Details Generic ANN Support ANN Training Data Cache Other Features A Summary of HTK-ANN HTK based Hybrid/Tandem Systems & Experiments Hybrid SI System Tandem SAT System Demo Hybrid System with Flexible Structures Conclusions 2of14

Design Principles The design should be as generic as possible. Flexible input feature configurations. Flexible ANN model architectures. HTK-ANN should be compatible with existing functions. To minimise the e ort to reuse previous source code and tools. To simplify the transfer of many technologies. HTK-ANN should be kept research friendly. 3of14

Generic ANN Support In HTK-ANN, ANNs have layered structures. An HMM set can have any number of ANNs. Each ANN can have any number of layers. An ANN layer has Parameters: weights, biases, activation function parameters An input vector: defined by a feature mixture structure A feature mixture has any number of feature elements A feature element defines a fragment of the input vector by Source: acoustic features, augmented features, output of some layer. A context shift set: integers indicated the time di erence. 4of14

Generic ANN Support In HTK-ANN, ANN structures can be any directed cyclic graph. Since only standard EBP is included at present, HTK-ANN can train non-recurrent ANNs properly (directed acyclic graph). t-6 t-3 t t+3 t+6 t t-1 t t+1 Feature Element 1 Feature Element 2 Feature Element 3 Source: Input acoustic features Context Shift Set: {-6, -3, 0, 3, 6} Source: ANN 1, Layer 3, Outputs Context Shift Set: {0} Source: ANN 2, Layer 2, Outputs Context Shift Set: {-1, 0, 1} Figure: An example of a feature mixture. 5of14

ANN Training HTK-ANN supports di erent training criteria Frame-level: CE, MMSE Sequence-level: MMI, MPE, MWE ANN model training labels can come from Frame-to-label alignment: for CE and MMSE criteria Feature files: for autoencoders Lattice files: for MMI, MPE, and MWE criteria Gradients for SGD can be modified with momentum, gradient clipping, weight decay, and max norm. Supported learning rate schedulers include List, Exponential Decay, AdaGrad, and a modified NewBob. 6of14

Data Cache HTK-ANN has three types of data shu ing Frame based shu ing: CE/MMSE for DNN, (unfolded) RNN Utterance based shu ing: MMI, MPE, and MWE training Batch of utterance level shu ing: RNN, ASGD 5 3 1 4 1 2 3 1 2 3 4 1 2 3 1 2 3 4 5 1 2 3 4 batch t batch t batch t Figure: Examples of di erent types of data shu ing. 7of14

Other Features Math Kernels: CPU, MKL, and CUDA based new kernels for ANNs Input Transforms: compatible with HTK SI/SD input transforms Speaker Adaptation: an ANN parameter unit online replacement Model Edit Insert/Remove/Initialise an ANN layer Add/Delete a feature element to a feature mixture Associate an ANN model to HMMs Decoders HVite: tandem/hybrid system decoding/alignment/model marking HDecode: tandem/hybrid system LVCSR decoding HDecode.mod: tandem/hybrid system model marking A Joint decoder: log-linear combination of systems (same decision tree) 8of14

A Summary of HTK-ANN Extended modules: HFBLat, HMath, HModel, HParm, HRec, HLVRec New modules HANNet: ANN structures & core algorithms HCUDA: CUDA based math kernel functions HNCache: Data cache for data random access Extended tools: HDecode, HDecode.mod, HHEd, HVite New tools HNForward: ANN evaluation & output generation HNTrainSGD: SGD based ANN training 9of14

Building Hybrid SI Systems Steps of building CE based SI CD-DNN-HMMs using HTK Produce desired tied state GMM-HMMs by decision tree tying (HHEd) Generate ANN-HMMs by replacing GMMs with an ANN (HHEd) Generate frame-to-state labels with a pre-trained system (HVite) Train ANN-HMMs based on CE (HNTrainSGD) Steps for CD-DNN-HMM MPE training Generate num./den. lattices (HLRescore & HDecode) Phone mark num./den. lattices (HVite or HDecode.mod) Perform MPE training (HNTrainSGD) 10 of 14

ANN Front-ends for GMM-HMMs ANNs can be used as GMM-HMM front-ends by using a feature mixture to define the composition of the GMM-HMM input vector. HTK can accomodate a tandem SAT system as a single system Mean and variance normalisations are treated as activation functions. SD parameters are replaceable according to speaker ids. Mean/Variance Normalisation Pitch Pitch PLP HLDA CMLLR PLP Bottleneck DNN STC Figure: A composite ANN as a Tandem SAT system front-end. 11 of 14

Standard BOLT System Results Hybrid DNN structure: 504 2000 4 1000 12000 Tandem DNN structure: 504 2000 4 1000 26 12000 System Criterion %WER Hybrid SI CE 34.5 Hybrid SI MPE 31.6 Tandem SAT MPE 33.2 Hybrid SI Tandem SAT MPE 31.0 Table: Performance of BOLT tandem and hybrid systems with standard configurations evaluated on dev 14. is the joint decoding with system dependent combination weights (1.0, 0.2). 12 of 14

WSJ Demo Systems with Flexible Structures Stacking MLPs: (468 + (n 1) 200) 1000 200 3000, n =1, 2,... Each MLP takes all previous BN features as input. The top MLP does not have a BN layer. System was trained with CE based discriminative pre-training and fine-tuning. Systems were trained with 15 hours Wall Street Journal (WSJ0). FNN %Accuracy %WER Num Train Held-out 65k dt 65k et 1 69.9 58.1 9.3 10.9 2 72.8 59.1 9.0 10.4 3 73.9 59.1 8.8 10.7 Table: Performance of the WSJ0 Demo Systems. 13 of 14

Conclusions HTK-ANN integrates native support of ANNs into HTK. HTK based GMM technologies can be directly applied to ANN-based systems. HTK-ANN can train FNNs with very flexible configurations Topologies equivalent to DAG Di erent activation functions Various input features Frame-level and sequence-level training criteria Experiments on 300h CTS task showed HTK can generate standard state-of-the-art tandem and hybrid systems. WSJ0 experiments showed HTK can build systems with flexible structures. HTK-ANN will be available with the release of HTK 3.5 in 2015. 14 of 14