cancel
Showing results for 
Search instead for 
Did you mean: 

EDB*Loader Tool Performance Timing Tests on Postgres Databases

EDB Team Member

So just to give a brief summary of this tool first; the EDB*Loader tool is a high-performance bulk data loader that provides an Oracle compatible interface for EDB Postgres Advanced Server. The EDB*Loader command line utility loads data from an input source, typically a file, into one or more tables using a subset of the parameters offered by Oracle SQL Loader

 

We went through an exercise testing out the EDB*Loader tool with different Postgres database AS versions {9.4, 9.6,10.1} to show the speed of this high-performance bulk data loader going from 5 million up to 100 million rows which were all under 6 mins load time. We automated the bulk load & captured timings; first doing a truncate of the table before each load & a bounce of the Database cluster after each load to avoid data being cached, to get accurate timings. Some other things to point out; we were on a docker image container ( version 18.06.1-ce-win73 ) on Centos 6 on our laptop PC (16GB RAM and CPU 2.7GHz speed), using default database memory configuration, no indexes on the table, & we used a conventional load method with no parallelism which still shows how fast these timings are with such a large amount of data loaded. We did see some timing differences around 35,45, & 50  million rows loaded but overall witnessed quite impressive load speeds. Also, on the 10.1 version we had archive log mode on in which we still were seeing fast load times. We thought this was a great story to tell showing the speed of our high-performance data loader tool and we have graphed out each load amounts vs. timings as well. We have also attached the EDB*Loader excel sheet where we captured the timings on a chart for viewing as well... In conclusion, as you can see, this linear graph between each version is pretty close to scale.

 

 

Rows Loaded 9.4 9.6 10.1
5000000 00:00:13.5 00:00:11.9 00:00:09.2
10000000 00:00:26.7 00:00:17.7 00:00:18.3
15000000 00:00:39.2 00:00:28.0 00:00:28.0
20000000 00:00:51.3 00:00:35.6 00:00:40.4
25000000 00:01:07.7 00:00:45.6 00:00:46.2
30000000 00:01:15.8 00:00:54.8 00:01:14.9
35000000 00:01:45.6 00:01:08.1 00:01:37.2
40000000 00:02:00.0 00:01:23.3 00:02:01.5
45000000 00:02:26.4 00:01:53.5 00:02:06.5
50000000 00:02:43.6 00:01:57.7 00:02:34.6
55000000 00:03:22.1 00:02:17.4 00:02:41.7
60000000 00:03:44.4 00:02:30.0 00:03:13.3
70000000 00:03:16.0 00:03:11.1 00:02:46.2
80000000 00:03:52.8 00:03:30.4 00:03:48.8
90000000 00:04:21.1 00:04:02.2 00:04:30.9
100000000 00:05:02.0 00:04:57.7 00:05:28.3

 EDBLDR Performance Timings 100mil.png