Domain-invariant Partial Least Squares (di-pls) Regression: A novel method for unsupervised and semi-supervised calibration model adaptation

Transcription

1 Domain-invariant Partial Least Squares (di-pls) Regression: A novel method for unsupervised and semi-supervised calibration model adaptation R. Nikzad-Langerodi W. Zellinger E. Lughofer T. Reischer 2 S. Saminger-Platz Department of Knowledge-Based Mathematical Systems Johannes Kepler University Linz, Austria Fuzzy Logic Laboratory, Linz-Hagenberg 2 Metadynea GmbH, Krems, Austria th Winter Symposium on Chemometrics Ramin Nikzad-Langerodi (JKU Linz, Austria) Domain-Invariant PLS WSC- / 8

2 Introduction/Motivation Ramin Nikzad-Langerodi (JKU Linz, Austria) Domain-Invariant PLS WSC- 2 / 8

3 Introduction/Motivation Ramin Nikzad-Langerodi (JKU Linz, Austria) Domain-Invariant PLS WSC- 2 / 8

4 Introduction/Motivation Instrument Standardization (Cargill Corn Data Set) X T =X T F ===== F=X T X S DS,PDS,GLSW,SST... Ramin Nikzad-Langerodi (JKU Linz, Austria) Domain-Invariant PLS WSC- 2 / 8

5 Introduction/Motivation Instrument Standardization (Cargill Corn Data Set) X T =X T F ===== F=X T X S DS,PDS,GLSW,SST... Why not align distributions implicitly? Ramin Nikzad-Langerodi (JKU Linz, Austria) Domain-Invariant PLS WSC- 2 / 8

6 Previous Work on Domain Adaptation Maximum Mean Discrepancy MMD(P, Q) = E XS P[φ(X S )] E XT Q[φ(X T )] H φ : X H Transfer Component Analysis (TCA) Pan et al. 20 min φ MMD s.t. Maximize Variance Scatter Component Analysis (SCA) Ghifary et al. 206 Requires non-linear Kernels to align higher order moments Deep/Transfer Learning Correlation Alignment (Corral) Sun et Saenko 206 Central Moment Discrepancy (CMD) Zellinger et al. 207 Unsupervised/Difficult to Optimize Ramin Nikzad-Langerodi (JKU Linz, Austria) Domain-Invariant PLS WSC- 3 / 8

7 Our Approach - Domain Regularization Domain Invariant Principle Component Analysis (PCA) min t,p tpt 2 F + λf (t S, t T ) }{{} Domain Regularizer Penalize Difference between Source and Target Distributions in LV Space Ramin Nikzad-Langerodi (JKU Linz, Austria) Domain-Invariant PLS WSC- 4 / 8

8 Our Approach - Domain Regularization Domain Invariant Principle Component Analysis (PCA) min t,p tpt 2 F + λf (t S, t T ) }{{} Domain Regularizer =0 {}}{ f (t S, t T ) = E[t S ] E[t T ] + E[tS] 2 E[tT 2 ] = n S pt X T S X S p n T pt X T T X T p Penalize Difference between Source and Target Distributions in LV Space Ramin Nikzad-Langerodi (JKU Linz, Austria) Domain-Invariant PLS WSC- 4 / 8

9 Our Approach - Domain Regularization Domain Invariant Principle Component Analysis (PCA) min t,p tpt 2 F + λf (t S, t T ) }{{} Domain Regularizer var = E[(X E[X ]) ] = E[X 2 ] }{{} 0 { =0 }} { f (t S, t T ) = E[t S ] E[t T ] + E[tS] 2 E[tT 2 ] µ = E[X ] = 0 }{{} local mean centering = n S pt X T S X S p n T pt X T T X T p Penalize Difference between Source and Target Distributions in LV Space Ramin Nikzad-Langerodi (JKU Linz, Austria) Domain-Invariant PLS WSC- 4 / 8

10 Domain-Invariant PCA L(t, p) = X tp T 2 F + λ n S pt X T S X S p n T pt X T T X T p p L = 0 Unconstrained Solution p T = tt X t T t [ I + ( λ 2t T t n S XT S X S n T XT T X T )] Identity Matrix (J J) (Deflated) Source and Target Covariance Matrices Ramin Nikzad-Langerodi (JKU Linz, Austria) Domain-Invariant PLS WSC- 5 / 8

11 Domain-Invariant PLS L(w) = X yw T 2 F + λ n S wt X T S X S w n T wt X T T X T w w L = 0 Unconstrained Solution w T = yt X y T y [ I + ( λ 2y T y n S XT S X S n T XT T X T )] Identity Matrix (J J) (Deflated) Source and Target Covariance Matrices Ramin Nikzad-Langerodi (JKU Linz, Austria) Domain-Invariant PLS WSC- 6 / 8

12 Proof of Concept λ = 0 λ = 00 Domain Regularization Aligns Covariance Structure of Source and Target Data Ramin Nikzad-Langerodi (JKU Linz, Austria) Domain-Invariant PLS WSC- 7 / 8

13 Let s Take a Closer Look Domain-Invariant PLS w T = yt X y T y [ I + ( λ 2y T y n S XT S X S n T XT T X T )] Ramin Nikzad-Langerodi (JKU Linz, Austria) Domain-Invariant PLS WSC- 8 / 8

14 Let s Take a Closer Look Domain-Invariant PLS w T = yt X y T y [ I + ( λ 2y T y n S XT S X S n T XT T X T )] X = X S Unsupervised Ramin Nikzad-Langerodi (JKU Linz, Austria) Domain-Invariant PLS WSC- 8 / 8

15 Let s Take a Closer Look Domain-Invariant PLS w T = yt X y T y [ I + ( λ 2y T y n S XT S X S n T XT T X T )] X = X S Unsupervised X = [X S ; X T ] Semi-Supervised Ramin Nikzad-Langerodi (JKU Linz, Austria) Domain-Invariant PLS WSC- 8 / 8

16 Let s Take a Closer Look Domain-Invariant PLS w T = yt X y T y [ I + ( λ 2y T y n S XT S X S n T XT T X T )] X = X S Unsupervised X = [X S ; X T ] Semi-Supervised X S, X T / X Ramin Nikzad-Langerodi (JKU Linz, Austria) Domain-Invariant PLS WSC- 8 / 8

17 How to Set λ Choosing λ too high aligns the Noise (Go for Pareto Optimal Point) Ramin Nikzad-Langerodi (JKU Linz, Austria) Domain-Invariant PLS WSC- 9 / 8

18 Component-Wise Model Selection Optimize λ i for i =,..., A LVs separately Largest effect usually for the first LV For NIR data 0 8 λ 0 9 λ >> 0 9 tends to shrink w T X T Xw Alternate between Optimization and Deflation Ramin Nikzad-Langerodi (JKU Linz, Austria) Domain-Invariant PLS WSC- 0 / 8

19 Case Study - Melamine Formaldehyde (MF) Condensation Monitoring of Condensation by FT-NIR Spectroscopy Recipe Changes often require Adaptation/Recalibration of PLS Models Ramin Nikzad-Langerodi (JKU Linz, Austria) Domain-Invariant PLS WSC- / 8

20 Case Study - Melamine Formaldehyde (MF) Condensation Results Unsupervised Adaptation - Unlabeled Data from 3 Batches of Different Recipe Scenario RMSECV d B (T S, T T ) RMSEP PLS di-pls PLS di-pls PLS di-pls di-pls (Best) n.s n.s n.s n.s n.s n.s n.s Improvement in Source (/2) and Target (6/2) Domain Ramin Nikzad-Langerodi (JKU Linz, Austria) Domain-Invariant PLS WSC- 2 / 8

21 Case Study - Melamine Formaldehyde (MF) Condensation Results Semi-Supervised Adaptation - Unlabeled Data from 3 Batches + 25 Labeled Samples Scenario RMSECV d B (T S, T T ) RMSEP PLS di-pls PLS di-pls PLS di-pls n.s n.s n.s n.s n.s Outperformance of Standard PLS with Calibration Set Augmentation in 0/2 Scenarios Ramin Nikzad-Langerodi (JKU Linz, Austria) Domain-Invariant PLS WSC- 3 / 8

22 Case Study - Melamine Formaldehyde (MF) Condensation Unsupervised Model Adaptation Can improve predictions in target domain if P(X S ) P(X T ) P(y X S ) P(y X T ) (Mismatch in Marginal Distributions) (Conditionals are Similar) Semi-Supervised Model Adaptation is required if P(X S ) P(X T ) P(y X S ) P(y X T ) (Mismatch in Marginal Distributions) (Conditionals are Different) How to find out remains an open question! Ramin Nikzad-Langerodi (JKU Linz, Austria) Domain-Invariant PLS WSC- 4 / 8

23 Summary and Conclusion Domain Invariant Extensions to PCA and PLS Implicit distribution alignment Unsupervised/Semi- Supervised Adaptation (Component-Wise) Model Selection Tested on Real-World FT-NIR Dataset Ramin Nikzad-Langerodi (JKU Linz, Austria) Domain-Invariant PLS WSC- 5 / 8

24 Open Problems and Future Perspectives Open Problems Convexity {}}{ f (t S, t T ) = w T ( X T S X S X T T X T )w f = S 0 S + = QΛ + Q T Numerical Problems no guarantee that d(t S, t T ) gets smaller as λ is increased Future Perspectives Extension to Multiple Domains (i.e. X, X 2,...,X d X d+ ) S Heterogeneous Transfer Learning/Data Integration (i.e. X S X T ) Ramin Nikzad-Langerodi (JKU Linz, Austria) Domain-Invariant PLS WSC- 6 / 8

25 Acknowledgments This work was funded by the Austrian research funding association (FFG) under the scope of the COMET programme within the research project Industrial Methods for Process Analytical Chemistry - From Measurement Technologies to Information Systems (impacts) (contract # ). This programme is promoted by BMVIT, BMWFW, the federal state of Upper Austria and the federal state of Lower Austria. Ramin Nikzad-Langerodi (JKU Linz, Austria) Domain-Invariant PLS WSC- 7 / 8

26 Thank You! Ramin Nikzad-Langerodi (JKU Linz, Austria) Domain-Invariant PLS WSC- 8 / 8