Rapid Upgrades With Pg_Migrator

Transcription

1 Rapid Upgrades With Pg_Migrator BRUCE MOMJIAN, ENTERPRISEDB November, 00 Abstract Pg_Migrator allows migration between major releases of Postgres without a data dump/reload. This presentation explains how pg_migrator works. Creative Commons Attribution License

2 Why Pg_Migrator Very fast upgrades Optionally no additional disk space Rapid Upgrades With Pg-Migrator

3 Other Upgrade Options dump/restore Slony Rapid Upgrades With Pg-Migrator

4 General Method New features often require system table changes The internal data format rarely changes pg_migrator installs new system tables while using data files from the previous Postgres version. Rapid Upgrades With Pg-Migrator 3

5 How It Works: Initial Setup Old Cluster New Cluster Rapid Upgrades With Pg-Migrator 4

6 Decouple New Clog Via Freezing Old Cluster New Cluster 3 6 Freeze X X Rapid Upgrades With Pg-Migrator

7 Transfer Clog and XID Old Cluster New Cluster controldata xid controldata Rapid Upgrades With Pg-Migrator 6

8 Get Schema Dump Old Cluster pg_dumpall - -schema New Cluster Rapid Upgrades With Pg-Migrator 7

9 Reserve TOAST OIDs Using Relfilenodes Old Cluster New Cluster This is necessary because heap references to TOAST tables contain the TOAST oids for easy loup. Rapid Upgrades With Pg-Migrator

10 Restore Schema In New Cluster Old Cluster pg_dumpall - -schema New Cluster Rapid Upgrades With Pg-Migrator

11 Connect TOAST Placeholders To the Proper Relations Old Cluster New Cluster X 47 Rapid Upgrades With Pg-Migrator 0

12 Copy User Heap/Index Files Old Cluster New Cluster Rapid Upgrades With Pg-Migrator

13 Complete Old Cluster New Cluster Rapid Upgrades With Pg-Migrator

14 How It Works: In Detail Check for cluster compatability locale encoding integer datetimes (default changed from.3 ->.4) Use pg_dumpall to dump old cluster schema (no data) Freeze all new cluster rows (remove reference to entries) Rename tablespaces to *_old New cluster uses old xid counter value (see freeze above) Set system table frozen xids to match the current xid Create new users/databases Rapid Upgrades With Pg-Migrator 3

15 Collect cluster information Install support functions that call internal backend functions Create placeholder files to reserve relfilenode file names Create schema in new cluster Adjust new cluster to use reserved relfilenode names Delete placeholder toast relfilenode files Remove new cluster toast tables Create new cluster toast table using reserved relfilenode Assign new toast tables with proper relfilenodes to relations Copy or link files from old cluster to new cluster Toast tables have the same relfilenodes as in the old cluster Warn about any remaining issues, like REINDEX requirements Rapid Upgrades With Pg-Migrator 4

16 Sample Run: Validation Performing consistency checks Checking old data directory /u/pgsql.old/data checking base checking global checking pg_ checking pg_multixact checking pg_subtrans checking pg_tblspc checking pg_twophase checking pg_xlog Checking new data directory /u/pgsql/data checking base checking global checking pg_ checking pg_multixact checking pg_subtrans checking pg_tblspc checking pg_twophase checking pg_xlog Checking binaries in old cluster (/u/pgsql.old/bin) checking postgres checking pg_ctl checking pg_dumpall checking psql Rapid Upgrades With Pg-Migrator

17 Sample Run: Validation Checking binaries in new cluster (/u/pgsql/bin) checking postgres checking pg_ctl checking pg_dumpall checking psql Starting postmaster to service old cluster waiting for postmaster to start Getting pg_database and pg_largeobject relfilenodes Checking for columns with user-defined composite types Checking for columns with user-defined array types Checking for columns with user-defined enum types Checking for /contrib/isn with bigint-passing mismatch Checking for invalid name user columns Checking for tsquery user columns Creating script to adjust sequences Creating catalog dump Splitting old dump file Stopping postmaster servicing old cluster Starting postmaster to service new cluster waiting for postmaster to start Checking for presence of required libraries Stopping postmaster servicing new cluster *Checks complete* Rapid Upgrades With Pg-Migrator 6

18 Preparing for Migration Preparing for migration Starting postmaster to service new cluster waiting for postmaster to start Analyzing all rows on the new cluster Freezing all rows on the new cluster Getting pg_database and pg_largeobject relfilenodes Stopping postmaster servicing new cluster If pg_migrator fails after this point, you must re-initdb the new cluster before continuing. You will also need to remove the ".old" suffix from old $PGDATA/global/pg_control and old tablespace directory names before continuing. Rapid Upgrades With Pg-Migrator 7

19 Sample Run: Migration Performing migration Adding ".old" suffix to old global/pg_control Renaming tablespaces to *.old Deleting old commit s Copying commit s Setting next transaction id for new cluster Resetting WAL archives Starting postmaster to service new cluster waiting for postmaster to start Setting frozenxid counters in new cluster Creating databases in new cluster Stopping postmaster servicing new cluster Creating placeholder relfiles for toast relations Starting postmaster to service new cluster waiting for postmaster to start Restoring database schema Adding support functions to new cluster Restoring relations to use fixed toast file names Rapid Upgrades With Pg-Migrator

20 Sample Run: Migration Removing support functions from new cluster Stopping postmaster servicing new cluster Restoring user relation files Setting next oid for new cluster Adjusting sequences Checking for tsvector user columns Checking for hash and gin indexes Checking for bpchar_pattern_ops indexes *Upgrade complete* Optimizer statistics and free space information are not transferred by pg_migrator, so consider running: vacuumdb --all --analyze on the newly-upgraded cluster. Rapid Upgrades With Pg-Migrator

21 Possible Post-.4 Data Format Changes Change heap page header, including bitmask tuple header, including bitmask data value format index page format TOAST page format Conversion Method none convert to new page format on read convert to new page format on read create old data type in new cluster reindex, or recreate index methods convert to new page format on read Rapid Upgrades With Pg-Migrator 0

22 Migration Timings The last duration is 44 seconds. Migration Method Minutes dump/restore dump with parallel restore 0.0 pg_migrator in copy mode 44.0 pg_migrator in link mode 0.7 Database size: 0GB, 0 tables Timings courtesy of Stefan Kaltenbrunner (mastermind on IRC) Rapid Upgrades With Pg-Migrator

23 Conclusion Rapid Upgrades With Pg-Migrator