Domain-invariant Partial Least Squares (di-pls) Regression: A novel method for unsupervised and semi-supervised calibration model adaptation R. Nikzad-Langerodi W. Zellinger E. Lughofer T. Reischer 2 S. Saminger-Platz Department of Knowledge-Based Mathematical Systems Johannes Kepler University Linz, Austria Fuzzy Logic Laboratory, Linz-Hagenberg 2 Metadynea GmbH, Krems, Austria th Winter Symposium on Chemometrics Ramin Nikzad-Langerodi (JKU Linz, Austria) Domain-Invariant PLS WSC- / 8
Introduction/Motivation Ramin Nikzad-Langerodi (JKU Linz, Austria) Domain-Invariant PLS WSC- 2 / 8
Introduction/Motivation Ramin Nikzad-Langerodi (JKU Linz, Austria) Domain-Invariant PLS WSC- 2 / 8
Introduction/Motivation Instrument Standardization (Cargill Corn Data Set) X T =X T F ===== F=X T X S DS,PDS,GLSW,SST... Ramin Nikzad-Langerodi (JKU Linz, Austria) Domain-Invariant PLS WSC- 2 / 8
Introduction/Motivation Instrument Standardization (Cargill Corn Data Set) X T =X T F ===== F=X T X S DS,PDS,GLSW,SST... Why not align distributions implicitly? Ramin Nikzad-Langerodi (JKU Linz, Austria) Domain-Invariant PLS WSC- 2 / 8
Previous Work on Domain Adaptation Maximum Mean Discrepancy MMD(P, Q) = E XS P[φ(X S )] E XT Q[φ(X T )] H φ : X H Transfer Component Analysis (TCA) Pan et al. 20 min φ MMD s.t. Maximize Variance Scatter Component Analysis (SCA) Ghifary et al. 206 Requires non-linear Kernels to align higher order moments Deep/Transfer Learning Correlation Alignment (Corral) Sun et Saenko 206 Central Moment Discrepancy (CMD) Zellinger et al. 207 Unsupervised/Difficult to Optimize Ramin Nikzad-Langerodi (JKU Linz, Austria) Domain-Invariant PLS WSC- 3 / 8
Our Approach - Domain Regularization Domain Invariant Principle Component Analysis (PCA) min t,p tpt 2 F + λf (t S, t T ) }{{} Domain Regularizer Penalize Difference between Source and Target Distributions in LV Space Ramin Nikzad-Langerodi (JKU Linz, Austria) Domain-Invariant PLS WSC- 4 / 8
Our Approach - Domain Regularization Domain Invariant Principle Component Analysis (PCA) min t,p tpt 2 F + λf (t S, t T ) }{{} Domain Regularizer =0 {}}{ f (t S, t T ) = E[t S ] E[t T ] + E[tS] 2 E[tT 2 ] = n S pt X T S X S p n T pt X T T X T p Penalize Difference between Source and Target Distributions in LV Space Ramin Nikzad-Langerodi (JKU Linz, Austria) Domain-Invariant PLS WSC- 4 / 8
Our Approach - Domain Regularization Domain Invariant Principle Component Analysis (PCA) min t,p tpt 2 F + λf (t S, t T ) }{{} Domain Regularizer var = E[(X E[X ]) ] = E[X 2 ] }{{} 0 { =0 }} { f (t S, t T ) = E[t S ] E[t T ] + E[tS] 2 E[tT 2 ] µ = E[X ] = 0 }{{} local mean centering = n S pt X T S X S p n T pt X T T X T p Penalize Difference between Source and Target Distributions in LV Space Ramin Nikzad-Langerodi (JKU Linz, Austria) Domain-Invariant PLS WSC- 4 / 8
Domain-Invariant PCA L(t, p) = X tp T 2 F + λ n S pt X T S X S p n T pt X T T X T p p L = 0 Unconstrained Solution p T = tt X t T t [ I + ( λ 2t T t n S XT S X S n T XT T X T )] Identity Matrix (J J) (Deflated) Source and Target Covariance Matrices Ramin Nikzad-Langerodi (JKU Linz, Austria) Domain-Invariant PLS WSC- 5 / 8
Domain-Invariant PLS L(w) = X yw T 2 F + λ n S wt X T S X S w n T wt X T T X T w w L = 0 Unconstrained Solution w T = yt X y T y [ I + ( λ 2y T y n S XT S X S n T XT T X T )] Identity Matrix (J J) (Deflated) Source and Target Covariance Matrices Ramin Nikzad-Langerodi (JKU Linz, Austria) Domain-Invariant PLS WSC- 6 / 8
Proof of Concept λ = 0 λ = 00 Domain Regularization Aligns Covariance Structure of Source and Target Data Ramin Nikzad-Langerodi (JKU Linz, Austria) Domain-Invariant PLS WSC- 7 / 8
Let s Take a Closer Look Domain-Invariant PLS w T = yt X y T y [ I + ( λ 2y T y n S XT S X S n T XT T X T )] Ramin Nikzad-Langerodi (JKU Linz, Austria) Domain-Invariant PLS WSC- 8 / 8
Let s Take a Closer Look Domain-Invariant PLS w T = yt X y T y [ I + ( λ 2y T y n S XT S X S n T XT T X T )] X = X S Unsupervised Ramin Nikzad-Langerodi (JKU Linz, Austria) Domain-Invariant PLS WSC- 8 / 8
Let s Take a Closer Look Domain-Invariant PLS w T = yt X y T y [ I + ( λ 2y T y n S XT S X S n T XT T X T )] X = X S Unsupervised X = [X S ; X T ] Semi-Supervised Ramin Nikzad-Langerodi (JKU Linz, Austria) Domain-Invariant PLS WSC- 8 / 8
Let s Take a Closer Look Domain-Invariant PLS w T = yt X y T y [ I + ( λ 2y T y n S XT S X S n T XT T X T )] X = X S Unsupervised X = [X S ; X T ] Semi-Supervised X S, X T / X Ramin Nikzad-Langerodi (JKU Linz, Austria) Domain-Invariant PLS WSC- 8 / 8
How to Set λ Choosing λ too high aligns the Noise (Go for Pareto Optimal Point) Ramin Nikzad-Langerodi (JKU Linz, Austria) Domain-Invariant PLS WSC- 9 / 8
Component-Wise Model Selection Optimize λ i for i =,..., A LVs separately Largest effect usually for the first LV For NIR data 0 8 λ 0 9 λ >> 0 9 tends to shrink w T X T Xw Alternate between Optimization and Deflation Ramin Nikzad-Langerodi (JKU Linz, Austria) Domain-Invariant PLS WSC- 0 / 8
Case Study - Melamine Formaldehyde (MF) Condensation Monitoring of Condensation by FT-NIR Spectroscopy Recipe Changes often require Adaptation/Recalibration of PLS Models Ramin Nikzad-Langerodi (JKU Linz, Austria) Domain-Invariant PLS WSC- / 8
Case Study - Melamine Formaldehyde (MF) Condensation Results Unsupervised Adaptation - Unlabeled Data from 3 Batches of Different Recipe Scenario RMSECV d B (T S, T T ) RMSEP PLS di-pls PLS di-pls PLS di-pls di-pls (Best) 562 568 2.30 0.0059 0.0037 2.76 2.47 2.45 562 86 2.34 2.8 0.026 0.09 3.29 3.28 n.s. 2.85 562 862 2.8 0.08 0.06 2.2 2.23 n.s. 2.29 568 562 2.30 n.s. 0.007 0.006 2.67 2.58 2.59 568 86 2.34 2.9 0.07 0.0 2.90 2.92 n.s. 2.58 568 862 2.2 0.08 0.06 2.45 2.38 2.37 86 562 2.5 0.020 0.09 3.53 3.29 3.4 86 568 2.64 2.47 0.03 0.02 3.23 3.0 3.04 86 862 2.32 0.049 0.04 3.0 2.8 2.69 862 562 2.22 0.023 0.022 3.60 3.52 n.s. 2.95 862 568 2.29 2.4 0.023 0.09 4.2 4. n.s. 4.03 862 86 2.3 0.057 0.054 4.92 4.90 n.s. 4.65 Improvement in Source (/2) and Target (6/2) Domain Ramin Nikzad-Langerodi (JKU Linz, Austria) Domain-Invariant PLS WSC- 2 / 8
Case Study - Melamine Formaldehyde (MF) Condensation Results Semi-Supervised Adaptation - Unlabeled Data from 3 Batches + 25 Labeled Samples Scenario RMSECV d B (T S, T T ) RMSEP PLS di-pls PLS di-pls PLS di-pls 562 568 2.44 2.7 0.8 0.3 2.69 2.64 n.s. 562 86 2.38 2.8 0. 0.02 3.6 2.68 562 862 2.40 2.25 0.42 0.03 2.57 2.22 568 562 2.34 2.26 n.s. 0.008 0.008 2.60 2.63 n.s. 568 86 2.40 2.40 n.s. 0.06 0.008 3.22 2.82 568 862 2.33 2.29 n.s. 0.40 0.04 2.48 2.32 86 562 2.65 2.44 0.29 0.08 3.39 3.04 86 568 2.90 2.67 0.36 0.26 3.24 2.93 86 862 2.65 2.44 0.39 0.06 2.96 2.75 862 562 2.8 2.52 0.45 0.0 3.27 2.83 862 568 2.25 2.0 0.60 0.34 2.65 2.3 862 86 2.70 2.43 0.9 0. 3.86 3.33 Outperformance of Standard PLS with Calibration Set Augmentation in 0/2 Scenarios Ramin Nikzad-Langerodi (JKU Linz, Austria) Domain-Invariant PLS WSC- 3 / 8
Case Study - Melamine Formaldehyde (MF) Condensation Unsupervised Model Adaptation Can improve predictions in target domain if P(X S ) P(X T ) P(y X S ) P(y X T ) (Mismatch in Marginal Distributions) (Conditionals are Similar) Semi-Supervised Model Adaptation is required if P(X S ) P(X T ) P(y X S ) P(y X T ) (Mismatch in Marginal Distributions) (Conditionals are Different) How to find out remains an open question! Ramin Nikzad-Langerodi (JKU Linz, Austria) Domain-Invariant PLS WSC- 4 / 8
Summary and Conclusion Domain Invariant Extensions to PCA and PLS Implicit distribution alignment Unsupervised/Semi- Supervised Adaptation (Component-Wise) Model Selection Tested on Real-World FT-NIR Dataset Ramin Nikzad-Langerodi (JKU Linz, Austria) Domain-Invariant PLS WSC- 5 / 8
Open Problems and Future Perspectives Open Problems Convexity {}}{ f (t S, t T ) = w T ( X T S X S X T T X T )w f = S 0 S + = QΛ + Q T Numerical Problems no guarantee that d(t S, t T ) gets smaller as λ is increased Future Perspectives Extension to Multiple Domains (i.e. X, X 2,...,X d X d+ ) S Heterogeneous Transfer Learning/Data Integration (i.e. X S X T ) Ramin Nikzad-Langerodi (JKU Linz, Austria) Domain-Invariant PLS WSC- 6 / 8
Acknowledgments This work was funded by the Austrian research funding association (FFG) under the scope of the COMET programme within the research project Industrial Methods for Process Analytical Chemistry - From Measurement Technologies to Information Systems (impacts) (contract # 843546). This programme is promoted by BMVIT, BMWFW, the federal state of Upper Austria and the federal state of Lower Austria. Ramin Nikzad-Langerodi (JKU Linz, Austria) Domain-Invariant PLS WSC- 7 / 8
Thank You! Ramin Nikzad-Langerodi (JKU Linz, Austria) Domain-Invariant PLS WSC- 8 / 8