Давайте рассмотрим ситуацию, в которой нам необходимо выполнить 100,000 записей в таблицу базы данных (далее – БД).
Рассмотрим примитивный сопособ:
Session session = sessionFactory.openSession();
Transaction transaction = session.beginTransaction();
for(int i = 0; i < 100_000; i++){
Developer developer = new Developer(/*Some parameters*/);
session.save(developer);
}
transaction.commit();
session.close();
На первый взгляд кажется, что мы получим 100_000 записей в нашу БД, но на практике мы получим OutOfMemoryException примерно в тот момент, когда попытаемся выполнить 50_000 – тысячную запись. Это вызвано тем, что Hibernate кэширует все сохраняемые объекты в кэш сессии.
Как же нам решить данную проблему?
Для этого нам необходимо использовать обработку пакетов (batch processing).
Например, мы говорим Hibernate, что хотим вставлять каждые 50 объектов, как единый пакет. Для этого нам необходимо установить hibernate.jdbc.atch_size на 50 и написать, примерно, такой кусок кода:
Session session = sessionFactory.openSession();
Transaction transaction = session.beginTransaction();
for(int i = 0; i < 100000; i++){
Developer developer = new Developer(/*Some parameters*/);
session.save(developer);
if(i % 50 == 0){
session.flush();
session.clear();
}
}
transaction.commit();
session.close();
Для понимания того, как это работает на практике – рассмотрим пример небольшого приложения.
Пример:
Исходный код проекта можно скачать по ЭТОЙ ССЫЛКЕ.
Шаг 1. Создаём таблицу в нашей БД
HIBRNATE_DEVELOPERS
CREATE TABLE HIBERNATE_DEVELOPERS (
id INT NOT NULL auto_increment,
FIRST_NAME VARCHAR(50) default NULL,
LAST_NAME VARCHAR(50) default NULL,
SPECIALTY VARCHAR(50) default NULL,
EXPERIENCE INT default NULL,
SALARY INT default NULL,
PRIMARY KEY (id)
);
Шаг 2 . Создадим POJO – класс
Developer.java
package net.proselyte.hibernate.batch;
public class Developer {
private int id;
private String firstName;
private String lastName;
private String specialty;
private int experience;
private int salary;
/**
* Default Constructor
*/
public Developer() {
}
/**
* Plain constructor
*/
public Developer(String firstName, String lastName, String specialty, int experience, int salary) {
this.firstName = firstName;
this.lastName = lastName;
this.specialty = specialty;
this.experience = experience;
this.salary = salary;
}
/**
* Getters and Setters
*/
public int getId() {
return id;
}
public void setId(int id) {
this.id = id;
}
public String getFirstName() {
return firstName;
}
public void setFirstName(String firstName) {
this.firstName = firstName;
}
public String getLastName() {
return lastName;
}
public void setLastName(String lastName) {
this.lastName = lastName;
}
public String getSpecialty() {
return specialty;
}
public void setSpecialty(String specialty) {
this.specialty = specialty;
}
public int getExperience() {
return experience;
}
public void setExperience(int experience) {
this.experience = experience;
}
public int getSalary() {
return salary;
}
public void setSalary(int salary) {
this.salary = salary;
}
/**
* toString method (optional)
*/
@Override
public String toString() {
return "id: " + id +
"\nFirst Name: " + firstName +
"\nLast Name: " + lastName +
"\nSpecialty: " + specialty +
"\nExperience: " + experience +
"\nSalary: " + salary + "\n";
}
}
Шаг 3. Создаём конфигруационные файлы
hibernate.cfg.xml
<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE hibernate-configuration SYSTEM
"http://www.hibernate.org/dtd/hibernate-configuration-3.0.dtd">
<hibernate-configuration>
<session-factory>
<property name="hibernate.dialect">
org.hibernate.dialect.MySQLDialect
</property>
<property name="hibernate.connection.driver_class">
com.mysql.jdbc.Driver
</property>
<!-- Assume PROSELYTE_TUTORIAL is the database name -->
<property name="hibernate.connection.url">
jdbc:mysql://localhost/ИМЯ_ВАШЕЙ_БАЗЫ_ДАННЫХ
</property>
<property name="hibernate.connection.username">
ВАШЕ_ИМЯ_ПОЛЬЗОВАТЕЛЯ
</property>
<property name="hibernate.connection.password">
ВАШ_ПАРОЛЬ
</property>
<!-- Specifying batch size -->
<property name="hibernate.jdbc.batch_size">
50
</property>
<!-- List of XML mapping files -->
<mapping resource="Developer.hbm.xml"/>
</session-factory>
</hibernate-configuration>
Developer.hbm.xml
<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE hibernate-mapping PUBLIC
"-//Hibernate/Hibernate Mapping DTD//EN"
"http://www.hibernate.org/dtd/hibernate-mapping-3.0.dtd">
<hibernate-mapping>
<class name="net.proselyte.hibernate.batch.Developer" table="HIBERNATE_DEVELOPERS">
<meta attribute="class-description">
This class contains developer details.
</meta>
<id name="id" type="int" column="id">
<generator class="native"/>
</id>
<property name="firstName" column="FIRST_NAME" type="string"/>
<property name="lastName" column="LAST_NAME" type="string"/>
<property name="specialty" column="SPECIALTY" type="string"/>
<property name="experience" column="EXPERIENCE" type="int"/>
<property name="salary" column="SALARY" type="int"/>
</class>
</hibernate-mapping>
Шаг 4. Создаём класс DeveloperRunner.java
DeveloperRunner.java
package net.proselyte.hibernate.batch;
import org.hibernate.Session;
import org.hibernate.SessionFactory;
import org.hibernate.Transaction;
import org.hibernate.cfg.Configuration;
public class DeveloperRunner {
private static SessionFactory sessionFactory;
public static void main(String[] args) {
sessionFactory = new Configuration().configure().buildSessionFactory();
DeveloperRunner developerRunner = new DeveloperRunner();
System.out.println("Adding 100,000 developer's records to the database...");
developerRunner.addDevelopers();
System.out.println("100,000 developer's records successfully added to the database...");
sessionFactory.close();
}
public void addDevelopers() {
Session session = sessionFactory.openSession();
Transaction transaction = null;
Integer developerId = null;
transaction = session.beginTransaction();
for (int i = 0; i < 100_000; i++) {
String firstName = "First Name " + i;
String lastName = "Last Name " + i;
String specialty = "Specialty " + i;
int experience = i;
int salary = i * 10;
Developer developer = new Developer(firstName, lastName, specialty, experience, salary);
session.save(developer);
if (i % 50 == 0) {
session.flush();
session.clear();
}
}
transaction.commit();
session.close();
}
}
Если всё было дслеано правильно, то в результате работы программы мыполучим, примерно, следующий результат:
/usr/lib/jvm/java-8-oracle/bin/java -Didea.launcher.port=7533 -Didea.launcher.bin.path=/home/proselyte/Programming/Soft/IntellijIdea/bin -Dfile.encoding=UTF-8 -classpath /usr/lib/jvm/java-8-oracle/jre/lib/management-agent.jar:/usr/lib/jvm/java-8-oracle/jre/lib/plugin.jar:/usr/lib/jvm/java-8-oracle/jre/lib/rt.jar:/usr/lib/jvm/java-8-oracle/jre/lib/jsse.jar:/usr/lib/jvm/java-8-oracle/jre/lib/charsets.jar:/usr/lib/jvm/java-8-oracle/jre/lib/jce.jar:/usr/lib/jvm/java-8-oracle/jre/lib/resources.jar:/usr/lib/jvm/java-8-oracle/jre/lib/deploy.jar:/usr/lib/jvm/java-8-oracle/jre/lib/jfxswt.jar:/usr/lib/jvm/java-8-oracle/jre/lib/javaws.jar:/usr/lib/jvm/java-8-oracle/jre/lib/jfr.jar:/usr/lib/jvm/java-8-oracle/jre/lib/ext/dnsns.jar:/usr/lib/jvm/java-8-oracle/jre/lib/ext/sunpkcs11.jar:/usr/lib/jvm/java-8-oracle/jre/lib/ext/sunec.jar:/usr/lib/jvm/java-8-oracle/jre/lib/ext/sunjce_provider.jar:/usr/lib/jvm/java-8-oracle/jre/lib/ext/jaccess.jar:/usr/lib/jvm/java-8-oracle/jre/lib/ext/nashorn.jar:/usr/lib/jvm/java-8-oracle/jre/lib/ext/localedata.jar:/usr/lib/jvm/java-8-oracle/jre/lib/ext/zipfs.jar:/usr/lib/jvm/java-8-oracle/jre/lib/ext/cldrdata.jar:/usr/lib/jvm/java-8-oracle/jre/lib/ext/jfxrt.jar:/home/proselyte/Programming/IdeaProjects/ProselyteTutorials/Hibernate/target/classes:/home/proselyte/.m2/repository/org/springframework/spring-core/4.1.1.RELEASE/spring-core-4.1.1.RELEASE.jar:/home/proselyte/.m2/repository/commons-logging/commons-logging/1.1.3/commons-logging-1.1.3.jar:/home/proselyte/.m2/repository/org/springframework/spring-web/4.1.1.RELEASE/spring-web-4.1.1.RELEASE.jar:/home/proselyte/.m2/repository/org/springframework/spring-aop/4.1.1.RELEASE/spring-aop-4.1.1.RELEASE.jar:/home/proselyte/.m2/repository/aopalliance/aopalliance/1.0/aopalliance-1.0.jar:/home/proselyte/.m2/repository/org/springframework/spring-beans/4.1.1.RELEASE/spring-beans-4.1.1.RELEASE.jar:/home/proselyte/.m2/repository/org/springframework/spring-context/4.1.1.RELEASE/spring-context-4.1.1.RELEASE.jar:/home/proselyte/.m2/repository/javax/servlet/servlet-api/2.5/servlet-api-2.5.jar:/home/proselyte/.m2/repository/org/springframework/spring-webmvc/4.1.1.RELEASE/spring-webmvc-4.1.1.RELEASE.jar:/home/proselyte/.m2/repository/org/springframework/spring-expression/4.1.1.RELEASE/spring-expression-4.1.1.RELEASE.jar:/home/proselyte/.m2/repository/org/springframework/integration/spring-integration-file/4.2.1.RELEASE/spring-integration-file-4.2.1.RELEASE.jar:/home/proselyte/.m2/repository/org/springframework/integration/spring-integration-core/4.2.1.RELEASE/spring-integration-core-4.2.1.RELEASE.jar:/home/proselyte/.m2/repository/org/springframework/spring-messaging/4.2.2.RELEASE/spring-messaging-4.2.2.RELEASE.jar:/home/proselyte/.m2/repository/org/springframework/retry/spring-retry/1.1.2.RELEASE/spring-retry-1.1.2.RELEASE.jar:/home/proselyte/.m2/repository/org/springframework/spring-tx/4.2.2.RELEASE/spring-tx-4.2.2.RELEASE.jar:/home/proselyte/.m2/repository/commons-io/commons-io/2.4/commons-io-2.4.jar:/home/proselyte/.m2/repository/org/hibernate/hibernate-core/5.1.0.Final/hibernate-core-5.1.0.Final.jar:/home/proselyte/.m2/repository/org/jboss/logging/jboss-logging/3.3.0.Final/jboss-logging-3.3.0.Final.jar:/home/proselyte/.m2/repository/org/hibernate/javax/persistence/hibernate-jpa-2.1-api/1.0.0.Final/hibernate-jpa-2.1-api-1.0.0.Final.jar:/home/proselyte/.m2/repository/org/javassist/javassist/3.20.0-GA/javassist-3.20.0-GA.jar:/home/proselyte/.m2/repository/antlr/antlr/2.7.7/antlr-2.7.7.jar:/home/proselyte/.m2/repository/org/apache/geronimo/specs/geronimo-jta_1.1_spec/1.1.1/geronimo-jta_1.1_spec-1.1.1.jar:/home/proselyte/.m2/repository/org/jboss/jandex/2.0.0.Final/jandex-2.0.0.Final.jar:/home/proselyte/.m2/repository/com/fasterxml/classmate/1.3.0/classmate-1.3.0.jar:/home/proselyte/.m2/repository/dom4j/dom4j/1.6.1/dom4j-1.6.1.jar:/home/proselyte/.m2/repository/xml-apis/xml-apis/1.0.b2/xml-apis-1.0.b2.jar:/home/proselyte/.m2/repository/org/hibernate/common/hibernate-commons-annotations/5.0.1.Final/hibernate-commons-annotations-5.0.1.Final.jar:/home/proselyte/.m2/repository/javassist/javassist/3.12.1.GA/javassist-3.12.1.GA.jar:/home/proselyte/.m2/repository/mysql/mysql-connector-java/5.1.38/mysql-connector-java-5.1.38.jar:/home/proselyte/Programming/Soft/IntellijIdea/lib/idea_rt.jar com.intellij.rt.execution.application.AppMain net.proselyte.hibernate.batch.DeveloperRunner
Feb 24, 2016 12:54:18 PM org.hibernate.Version logVersion
INFO: HHH000412: Hibernate Core {5.1.0.Final}
Feb 24, 2016 12:54:18 PM org.hibernate.cfg.Environment
INFO: HHH000206: hibernate.properties not found
Feb 24, 2016 12:54:18 PM org.hibernate.cfg.Environment buildBytecodeProvider
INFO: HHH000021: Bytecode provider name : javassist
Feb 24, 2016 12:54:18 PM org.hibernate.annotations.common.reflection.java.JavaReflectionManager
INFO: HCANN000001: Hibernate Commons Annotations {5.0.1.Final}
Feb 24, 2016 12:54:20 PM org.hibernate.engine.jdbc.connections.internal.DriverManagerConnectionProviderImpl configure
WARN: HHH10001002: Using Hibernate built-in connection pool (not for production use!)
Feb 24, 2016 12:54:20 PM org.hibernate.engine.jdbc.connections.internal.DriverManagerConnectionProviderImpl buildCreator
INFO: HHH10001005: using driver [com.mysql.jdbc.Driver] at URL [jdbc:mysql://localhost/PROSELYTE_TUTORIAL]
Feb 24, 2016 12:54:20 PM org.hibernate.engine.jdbc.connections.internal.DriverManagerConnectionProviderImpl buildCreator
INFO: HHH10001001: Connection properties: {user=root, password=****}
Feb 24, 2016 12:54:20 PM org.hibernate.engine.jdbc.connections.internal.DriverManagerConnectionProviderImpl buildCreator
INFO: HHH10001003: Autocommit mode: false
Feb 24, 2016 12:54:20 PM org.hibernate.engine.jdbc.connections.internal.PooledConnections
INFO: HHH000115: Hibernate connection pool size: 20 (min=1)
Wed Feb 24 12:54:20 EET 2016 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Feb 24, 2016 12:54:20 PM org.hibernate.dialect.Dialect
INFO: HHH000400: Using dialect: org.hibernate.dialect.MySQLDialect
Adding 100,000 developer's records to the database...
100,000 developer's records successfully added to the database...
Feb 24, 2016 12:54:41 PM org.hibernate.engine.jdbc.connections.internal.DriverManagerConnectionProviderImpl stop
INFO: HHH10001008: Cleaning up connection pool [jdbc:mysql://localhost/PROSELYTE_TUTORIAL]
Примерно, вот так будет выглядеть наша таблица HIBERNATE_DEVELOPERS
+--------+------------------+-----------------+-----------------+------------+--------+
| id | FIRST_NAME | LAST_NAME | SPECIALTY | EXPERIENCE | SALARY |
+--------+------------------+-----------------+-----------------+------------+--------+
| 100061 | First Name 99988 | Last Name 99988 | Specialty 99988 | 99988 | 999880 |
| 100062 | First Name 99989 | Last Name 99989 | Specialty 99989 | 99989 | 999890 |
| 100063 | First Name 99990 | Last Name 99990 | Specialty 99990 | 99990 | 999900 |
| 100064 | First Name 99991 | Last Name 99991 | Specialty 99991 | 99991 | 999910 |
| 100065 | First Name 99992 | Last Name 99992 | Specialty 99992 | 99992 | 999920 |
| 100066 | First Name 99993 | Last Name 99993 | Specialty 99993 | 99993 | 999930 |
| 100067 | First Name 99994 | Last Name 99994 | Specialty 99994 | 99994 | 999940 |
| 100068 | First Name 99995 | Last Name 99995 | Specialty 99995 | 99995 | 999950 |
| 100069 | First Name 99996 | Last Name 99996 | Specialty 99996 | 99996 | 999960 |
| 100070 | First Name 99997 | Last Name 99997 | Specialty 99997 | 99997 | 999970 |
| 100071 | First Name 99998 | Last Name 99998 | Specialty 99998 | 99998 | 999980 |
| 100072 | First Name 99999 | Last Name 99999 | Specialty 99999 | 99999 | 999990 |
+--------+------------------+-----------------+-----------------+------------+--------+
В этой статье мы ознакомились с понятием обработки пакетов (batch processing), и рассмотрели пример приложения с её применением.