Skip to content

Commit 29c55de

Browse files
bytefishbytefish
committed
Update README.md
1 parent 81a1c5a commit 29c55de

1 file changed

Lines changed: 71 additions & 274 deletions

File tree

‎README.md‎

Lines changed: 71 additions & 274 deletions
Original file line numberDiff line numberDiff line change
@@ -1,319 +1,116 @@
1-
[![Maven Central](https://img.shields.io/maven-central/v/de.bytefish/jsqlserverbulkinsert.svg?label=Maven%20Central)](https://search.maven.org/artifact/de.bytefish/jsqlserverbulkinsert)
2-
# JSqlServerBulkInsert #
1+
# JSqlServerBulkInsert: High-Performance Bulk Inserts to SQL Server #
32

4-
[JSqlServerBulkInsert]: https://github.com/bytefish/JSqlServerBulkInsert
5-
[MIT License]: https://opensource.org/licenses/MIT
3+
JSqlServerBulkInsert is a high-performance Java library for Bulk Inserts to Microsoft
4+
SQL Server using the native `ISQLServerBulkData` API.
65

7-
[JSqlServerBulkInsert] is a library to simplify Bulk Inserts to the SQL Server. It wraps the ``SQLServerBulkCopy`` behind a nice API.
6+
It provides an elegant, functional wrapper around the SQL Server Bulk Copy functionality:
87

9-
## Installing ##
8+
> Bulk copy is a feature that allows efficient bulk import of data into a SQL Server table. It
9+
> is significantly faster than using standard `INSERT` statements, especially for large datasets.
1010
11-
You can obtain [JSqlServerBulkInsert] from Maven by adding the following:
11+
## Setup ##
12+
13+
JSqlServerBulkInsert is designed for Java 17+ and the Microsoft JDBC Driver for SQL Server (version 8.1.0 or higher).
14+
15+
Add the following dependency to your pom.xml:
1216

1317
```xml
1418
<dependency>
15-
<groupId>de.bytefish</groupId>
16-
<artifactId>jsqlserverbulkinsert</artifactId>
17-
<version>5.1.0</version>
19+
<groupId>de.bytefish</groupId>
20+
<artifactId>jsqlserverbulkinsert</artifactId>
21+
<version>6.0.0</version>
1822
</dependency>
1923
```
2024

21-
## Supported Types ##
22-
23-
Please read up the Microsoft Documentation for understanding the mapping between SQL Server Types and JDBC Data Types:
25+
## A Re-Designed API for 6.0.0 ##
2426

25-
* [Understanding the JDBC Driver Data Types](https://docs.microsoft.com/en-us/sql/connect/jdbc/understanding-the-jdbc-driver-data-types)
27+
Version 6.0.0 introduces a completely redesigned API. It strictly separates
28+
the **What** (Structure and Mapping) from the **How** (Execution and I/O).
2629

27-
The following JDBC Types are supported by the library:
30+
### Key Features: ###
2831

29-
* Numeric Types
30-
* TINYINT
31-
* SMALLINT
32-
* INTEGER
33-
* BIGINT
34-
* NUMERIC
35-
* REAL
36-
* DOUBLE
37-
* Date/Time Types
38-
* DATE
39-
* TIMESTAMP
40-
* TIMESTAMP with Timezone
41-
* Boolean Type
42-
* BIT
43-
* Character / Text Types
44-
* CHAR
45-
* NCHAR
46-
* CLOB
47-
* VARCHAR
48-
* NVARCHAR
49-
* LONGVARCHAR
50-
* NLONGVARCHAR
51-
* Binary Data Types
52-
* VARBINARY
53-
54-
55-
## Notes on the Table Mapping ##
32+
* Stateless Mapping: Define your schema once and reuse it across multiple writers.
33+
* Primitive Support: Specialized functional interfaces for primitive types (int, long, boolean, float, double) simplify the mapping process.
34+
* Automated Bracketing: Automatic [] escaping for all schema, table, and column names to prevent keyword conflicts (e.g., with columns named `LOCALTIME` or `DATE`).
35+
* Modern Time API: Native support for java.time types with optimized binary transfer.
5636

57-
The ``SQLServerBulkCopy`` implementation of Microsoft requires **all columns** of the destination table
58-
to be defined, even if the columns contain auto-generated values.
37+
## Quick Start ##
5938

60-
So imagine you have a table with an auto-incrementing primary key:
39+
### 1. Define your Data Model ###
6140

62-
```sql
63-
CREATE TABLE [dbo].[UnitTest](
64-
PK_ID INT IDENTITY(1,1) PRIMARY KEY,
65-
IntegerValue INT
66-
)
67-
```
68-
69-
And although our class doesn't contain the Auto-Incrementing Primary Key:
41+
The library works perfectly with modern Java record types or traditional POJOs.
7042

7143
```java
72-
public class MySampleEntity {
73-
74-
private final int val;
75-
76-
public MySampleEntity(int val) {
77-
this.val = val;
78-
}
79-
80-
public Integer getVal() {
81-
return val;
82-
}
83-
}
44+
public record SensorData(
45+
UUID id,
46+
String name,
47+
int temperature,
48+
double signal,
49+
OffsetDateTime timestamp
50+
) {}
8451
```
8552

86-
We still need to map it in the AbstractMapping like this:
53+
### 2. Define your Mapping (Stateless & Thread-Safe) ###
8754

88-
```java
89-
public class MySampleEntityMapping extends AbstractMapping<MySampleEntity> {
90-
91-
public MySampleEntityMapping() {
92-
super("dbo", "UnitTest");
93-
94-
mapInteger("PK_ID", x -> null);
95-
mapInteger("IntegerValue", x -> x.getVal());
96-
}
97-
}
98-
```
99-
100-
Or like this to explicitly define the Column as auto-incrementing:
55+
The `SqlServerMapper<T>` is the heart of the library. It is stateless after configuration and should
56+
be instantiated only once (e.g., as a `static final` field).
10157

10258
```java
103-
private class MySampleEntityMapping extends AbstractMapping<MySampleEntity> {
104-
105-
public MySampleEntityMapping() {
106-
super("dbo", "UnitTest");
107-
108-
mapInteger("PK_ID", true);
109-
mapInteger("IntegerValue", x -> x.getVal());
110-
}
111-
}
59+
private static final SqlServerMapper<SensorData> MAPPER =
60+
SqlServerMapper.forClass(SensorData.class)
61+
.map("Id", SqlServerTypes.UNIQUEIDENTIFIER.from(SensorData::id))
62+
63+
// Use specialized primitive extractors for better readability
64+
.map("Temperature", SqlServerTypes.INT.primitive(SensorData::temperature))
65+
.map("SignalStrength", SqlServerTypes.FLOAT.primitive(SensorData::signal))
66+
67+
.map("Name", SqlServerTypes.NVARCHAR.from(SensorData::name))
68+
69+
// TIME TYPES: Optimized binary transfer via native precision metadata
70+
.map("Timestamp", SqlServerTypes.DATETIMEOFFSET.offsetDateTime(SensorData::timestamp));
11271
```
11372

114-
### Notes on DATETIME Columns ###
115-
116-
If you are trying to map a `LocalDateTime` to a `DATETIME` column, you need to drop the nanoseconds part. A SQL Server `DATETIME` column doesn't have this level of precision.
73+
### 3. Execute the Bulk Insert ###
11774

118-
It can be fixed by using `LocalDateTime#truncatedTo(ChronoUnit.MILLIS)`, like this:
75+
The `SqlServerBulkWriter<T>` is a lightweight, transient executor that streams the data to the database.
11976

12077
```java
121-
public class Issue21EntityMapping extends AbstractMapping<Issue21Entity> {
78+
public void saveAll(Connection conn, List<SensorData> data) {
12279

123-
public Issue21EntityMapping() {
124-
super("dbo", "UnitTest");
125-
126-
mapLocalDateTime("LastUpdated", x -> x.getLastUpdate().truncatedTo(ChronoUnit.MILLIS));
80+
SqlServerBulkWriter<SensorData> writer = new SqlServerBulkWriter<>(MAPPER)
81+
.withBatchSize(1000)
82+
.withTableLock(true);
83+
84+
BulkInsertResult result = writer.saveAll(conn, "dbo", "Sensors", data);
85+
86+
if (result.success()) {
87+
System.out.println("Inserted " + result.rowsAffected() + " rows.");
12788
}
12889
}
12990
```
13091

131-
### Order of Columns ###
132-
133-
The ``SqlServerBulkCopy`` implementation of the Microsoft JDBC driver requires, that the destination schema and
134-
mapping have same column order. This is done automatically by querying the metadata of the table and sorting your
135-
mappings, before inserting the data.
136-
137-
If this cannot be done automatically, because the JDBC driver does not return the metadata, **then the mapping and
138-
destination schema have to match, and the fields must be mapped in the same order as the destination table**.
92+
## Mastering the Fluent API ##
13993

140-
## Getting Started ##
94+
### Monitoring Progress ###
14195

142-
Imagine ``1,000,000`` Persons should be inserted into an SQL Server database.
143-
144-
### Results ###
145-
146-
Bulk Inserting ``1,000,000``entities to a SQL Server 2016 database took ``5`` Seconds:
147-
148-
```
149-
[Bulk Insert 1000000 Entities] PT4.559S
150-
```
151-
### Domain Model ###
152-
153-
The domain model could be the ``Person`` class with a First Name, Last Name and a birth date.
96+
You can monitor the progress of long-running bulk inserts:
15497

15598
```java
156-
package de.bytefish.jsqlserverbulkinsert.test.model;
157-
158-
import java.time.LocalDate;
159-
160-
public class Person {
161-
162-
private String firstName;
163-
164-
private String lastName;
165-
166-
private LocalDate birthDate;
167-
168-
public Person() {
169-
}
170-
171-
public String getFirstName() {
172-
return firstName;
173-
}
174-
175-
public void setFirstName(String firstName) {
176-
this.firstName = firstName;
177-
}
178-
179-
public String getLastName() {
180-
return lastName;
181-
}
182-
183-
public void setLastName(String lastName) {
184-
this.lastName = lastName;
185-
}
186-
187-
public LocalDate getBirthDate() {
188-
return birthDate;
189-
}
190-
191-
public void setBirthDate(LocalDate birthDate) {
192-
this.birthDate = birthDate;
193-
}
194-
}
195-
```
196-
197-
### Mapping ###
198-
199-
To bulk insert the ``Person`` data to a SQL Server database it is important to know how to map
200-
between the Java Object and the Database Columns:
201-
202-
```java
203-
package de.bytefish.jsqlserverbulkinsert.test.integration;
204-
205-
import de.bytefish.jsqlserverbulkinsert.mapping.AbstractMapping;
206-
import de.bytefish.jsqlserverbulkinsert.test.model.Person;
207-
208-
public class PersonMapping extends AbstractMapping<Person> {
209-
210-
public PersonMapping() {
211-
super("dbo", "UnitTest");
212-
213-
mapNvarchar("FirstName", Person::getFirstName);
214-
mapNvarchar("LastName", Person::getLastName);
215-
mapDate("BirthDate", Person::getBirthDate);
216-
}
217-
}
99+
writer.withNotifyAfter(1000, rows -> System.out.println("Processed " + rows + " rows."));
218100
```
219101

220-
### Construct and use the SqlServerBulkInsert ###
102+
### Error Handling ###
221103

222-
The ``AbstractMapping`` is used to instantiate a ``SqlServerBulkInsert``, which provides
223-
a ``saveAll`` method to store a given stream of data.
104+
Catch and handle specific SQL Server errors through a dedicated handler:
224105

225106
```java
226-
// Instantiate the SqlServerBulkInsert class:
227-
SqlServerBulkInsert<Person> bulkInsert = new SqlServerBulkInsert<>(mapping);
228-
// Now save all entities of a given stream:
229-
bulkInsert.saveAll(connection, persons.stream());
107+
writer.withErrorHandler(ex -> log.error("Bulk Insert failed: " + ex.getMessage()));
230108
```
231109

232-
And the full Integration Test:
233-
234-
```java
235-
package de.bytefish.jsqlserverbulkinsert.test.integration;
236-
237-
import de.bytefish.jsqlserverbulkinsert.SqlServerBulkInsert;
238-
import de.bytefish.jsqlserverbulkinsert.test.base.TransactionalTestBase;
239-
import de.bytefish.jsqlserverbulkinsert.test.model.Person;
240-
import de.bytefish.jsqlserverbulkinsert.test.utils.MeasurementUtils;
241-
import org.junit.Assert;
242-
import org.junit.Test;
243-
244-
import java.sql.ResultSet;
245-
import java.sql.SQLException;
246-
import java.sql.Statement;
247-
import java.time.LocalDate;
248-
import java.util.ArrayList;
249-
import java.util.List;
110+
## Supported SQL Server Types ##
250111

251-
public class IntegrationTest extends TransactionalTestBase {
252-
253-
@Override
254-
protected void onSetUpInTransaction() throws Exception {
255-
createTable();
256-
}
257-
258-
@Test
259-
public void bulkInsertPersonDataTest() throws SQLException {
260-
// The Number of Entities to insert:
261-
int numEntities = 1000000;
262-
// Create a large list of Persons:
263-
List<Person> persons = getPersonList(numEntities);
264-
// Create the Mapping:
265-
PersonMapping mapping = new PersonMapping();
266-
// Create the Bulk Inserter:
267-
SqlServerBulkInsert<Person> bulkInsert = new SqlServerBulkInsert<>(mapping);
268-
// Measure the Bulk Insert time:
269-
MeasurementUtils.MeasureElapsedTime("Bulk Insert 1000000 Entities", () -> {
270-
// Now save all entities of a given stream:
271-
bulkInsert.saveAll(connection, persons.stream());
272-
});
273-
// And assert all have been written to the database:
274-
Assert.assertEquals(numEntities, getRowCount());
275-
}
276-
277-
private List<Person> getPersonList(int numPersons) {
278-
List<Person> persons = new ArrayList<>();
279-
280-
for (int pos = 0; pos < numPersons; pos++) {
281-
Person p = new Person();
282-
283-
p.setFirstName("Philipp");
284-
p.setLastName("Wagner");
285-
p.setBirthDate(LocalDate.of(1986, 5, 12));
286-
287-
persons.add(p);
288-
}
289-
290-
return persons;
291-
}
292-
293-
private boolean createTable() throws SQLException {
294-
295-
String sqlStatement = "CREATE TABLE [dbo].[UnitTest]\n" +
296-
" (\n" +
297-
" FirstName NVARCHAR(255),\n" +
298-
" LastName NVARCHAR(255),\n" +
299-
" BirthDate DATE\n" +
300-
" );";
301-
302-
Statement statement = connection.createStatement();
303-
304-
return statement.execute(sqlStatement);
305-
}
306-
307-
private int getRowCount() throws SQLException {
308-
309-
Statement s = connection.createStatement();
310-
311-
ResultSet r = s.executeQuery("SELECT COUNT(*) AS total FROM [dbo].[UnitTest];");
312-
r.next();
313-
int count = r.getInt("total");
314-
r.close();
315-
316-
return count;
317-
}
318-
}
319-
```
112+
* Numeric Types: `BIT` (Boolean), `TINYINT`, `SMALLINT`, `INT`, `BIGINT`, `REAL`, `FLOAT` (Double), `NUMERIC`, `DECIMAL`, `MONEY`, `SMALLMONEY`
113+
* Character Types: `CHAR`, `VARCHAR`, `NCHAR`, `NVARCHAR` (including `.max()` support)
114+
* Temporal Types: `DATE`, `TIME`, `DATETIME`, `DATETIME2`, `SMALLDATETIME`, `DATETIMEOFFSET`
115+
* Binary Types: `VARBINARY` (including `.max()` support)
116+
* Other Types: `UNIQUEIDENTIFIER` (UUID)

0 commit comments

Comments
 (0)