Skip to content

Commit 73faf9d

Browse files
committed
Update README.md
1 parent a5ae84a commit 73faf9d

1 file changed

Lines changed: 0 additions & 322 deletions

File tree

README.md

Lines changed: 0 additions & 322 deletions
Original file line numberDiff line numberDiff line change
@@ -212,325 +212,3 @@ Native support for all PostgreSQL geometric types using dedicated helper records
212212
* path
213213
* polygon
214214
* circle
215-
216-
217-
## Usage ##
218-
219-
You can use the [PgBulkInsert] API in various ways. The first one is to use the ``SimpleRowWriter`` when you don't have
220-
an explicit Java POJO, that matches a Table. The second way is to use an ``AbstractMapping<TEntityType>`` to define a
221-
mapping between a Java POJO and a PostgreSQL table.
222-
223-
Please also read the FAQ, which may answer some of your questions.
224-
225-
## Using the SimpleRowWriter ##
226-
227-
Using the ``SimpleRowWriter`` doesn't require you to define a separate mapping. It requires you to define the PostgreSQL table structure using
228-
a ``SimpleRowWriter.Table``, that has a schema name (optional), table name and column names:
229-
230-
```java
231-
// Schema of the Table:
232-
String schemaName = "sample";
233-
234-
// Name of the Table:
235-
String tableName = "row_writer_test";
236-
237-
// Define the Columns to be inserted:
238-
String[] columnNames = new String[] {
239-
"value_int",
240-
"value_text"
241-
};
242-
243-
// Create the Table Definition:
244-
SimpleRowWriter.Table table = new SimpleRowWriter.Table(schemaName, tableName, columnNames);
245-
```
246-
247-
Once created you create the ``SimpleRowWriter`` by using the ``Table`` and a ``PGConnection``.
248-
249-
Now to write a row to PostgreSQL you call the ``startRow`` method. It expects you to pass a
250-
``Consumer<SimpleRow>`` into it, which defines what data to write to the row. The call to
251-
``startRow`` is synchronized, so it is safe to be called from multiple threads.
252-
253-
```java
254-
// Create the Writer:
255-
try(SimpleRowWriter writer = new SimpleRowWriter(table, pgConnection)) {
256-
257-
// ... write your data rows:
258-
for(int rowIdx = 0; rowIdx < 10000; rowIdx++) {
259-
260-
// ... using startRow and work with the row, see how the order doesn't matter:
261-
writer.startRow((row) -> {
262-
row.setText("value_text", "Hi");
263-
row.setInteger("value_int", 1);
264-
});
265-
}
266-
}
267-
```
268-
269-
So the complete example looks like this:
270-
271-
```java
272-
public class SimpleRowWriterTest extends TransactionalTestBase {
273-
274-
// ...
275-
276-
@Test
277-
public void rowBasedWriterTest() throws SQLException {
278-
279-
// Get the underlying PGConnection:
280-
PGConnection pgConnection = PostgreSqlUtils.getPGConnection(connection);
281-
282-
// Schema of the Table:
283-
String schemaName = "sample";
284-
285-
// Name of the Table:
286-
String tableName = "row_writer_test";
287-
288-
// Define the Columns to be inserted:
289-
String[] columnNames = new String[] {
290-
"value_int",
291-
"value_text"
292-
};
293-
294-
// Create the Table Definition:
295-
SimpleRowWriter.Table table = new SimpleRowWriter.Table(schemaName, tableName, columnNames);
296-
297-
// Create the Writer:
298-
try(SimpleRowWriter writer = new SimpleRowWriter(table, pgConnection)) {
299-
300-
// ... write your data rows:
301-
for(int rowIdx = 0; rowIdx < 10000; rowIdx++) {
302-
303-
// ... using startRow and work with the row, see how the order doesn't matter:
304-
writer.startRow((row) -> {
305-
row.setText("value_text", "Hi");
306-
row.setInteger("value_int", 1);
307-
});
308-
309-
}
310-
}
311-
312-
// Now assert, that we have written 10000 entities:
313-
314-
Assert.assertEquals(10000, getRowCount());
315-
}
316-
}
317-
```
318-
319-
If you need to customize the Null Character Handling, then you can use the ``setNullCharacterHandler(Function<String, String> nullCharacterHandler)`` function.
320-
321-
## Using the AbstractMapping ##
322-
323-
The ``AbstractMapping`` is the second possible way to map a POJO for usage in PgBulkInsert. Imagine we want to bulk insert a large amount of people
324-
into a PostgreSQL database. Each ``Person`` has a first name, a last name and a birthdate.
325-
326-
### Database Table ###
327-
328-
The table in the PostgreSQL database might look like this:
329-
330-
```sql
331-
CREATE TABLE sample.unit_test
332-
(
333-
first_name text,
334-
last_name text,
335-
birth_date date
336-
);
337-
```
338-
339-
### Domain Model ###
340-
341-
The domain model in the application might look like this:
342-
343-
```java
344-
private class Person {
345-
346-
private String firstName;
347-
348-
private String lastName;
349-
350-
private LocalDate birthDate;
351-
352-
public Person() {}
353-
354-
public String getFirstName() {
355-
return firstName;
356-
}
357-
358-
public void setFirstName(String firstName) {
359-
this.firstName = firstName;
360-
}
361-
362-
public String getLastName() {
363-
return lastName;
364-
}
365-
366-
public void setLastName(String lastName) {
367-
this.lastName = lastName;
368-
}
369-
370-
public LocalDate getBirthDate() {
371-
return birthDate;
372-
}
373-
374-
public void setBirthDate(LocalDate birthDate) {
375-
this.birthDate = birthDate;
376-
}
377-
378-
}
379-
```
380-
381-
### Bulk Inserter ###
382-
383-
Then you have to implement the ``AbstractMapping<Person>``, which defines the mapping between the table and the domain model.
384-
385-
```java
386-
public class PersonMapping extends AbstractMapping<Person>
387-
{
388-
public PersonMapping() {
389-
super("sample", "unit_test");
390-
391-
mapText("first_name", Person::getFirstName);
392-
mapText("last_name", Person::getLastName);
393-
mapDate("birth_date", Person::getBirthDate);
394-
}
395-
}
396-
```
397-
398-
This mapping is used to create the ``PgBulkInsert<Person>``:
399-
400-
```java
401-
PgBulkInsert<Person> bulkInsert = new PgBulkInsert<Person>(new PersonMapping());
402-
```
403-
404-
### Using the Bulk Inserter ###
405-
406-
[IntegrationTest.java]: https://github.com/bytefish/PgBulkInsert/blob/master/PgBulkInsert/pgbulkinsert-core/src/test/java/de/bytefish/pgbulkinsert/integration/IntegrationTest.java
407-
408-
And finally we can write a Unit Test to insert ``100000`` people into the database. You can find the entire Unit Test on GitHub as [IntegrationTest.java].
409-
410-
```java
411-
@Test
412-
public void bulkInsertPersonDataTest() throws SQLException {
413-
// Create a large list of People:
414-
List<Person> personList = getPersonList(100000);
415-
// Create the BulkInserter:
416-
PgBulkInsert<Person> bulkInsert = new PgBulkInsert<Person>(new PersonMapping(schema));
417-
// Now save all entities of a given stream:
418-
bulkInsert.saveAll(PostgreSqlUtils.getPGConnection(connection), personList.stream());
419-
// And assert all have been written to the database:
420-
Assert.assertEquals(100000, getRowCount());
421-
}
422-
423-
private List<Person> getPersonList(int num) {
424-
List<Person> personList = new ArrayList<>();
425-
426-
for (int pos = 0; pos < num; pos++) {
427-
Person p = new Person();
428-
429-
p.setFirstName("Philipp");
430-
p.setLastName("Wagner");
431-
p.setBirthDate(LocalDate.of(1986, 5, 12));
432-
433-
personList.add(p);
434-
}
435-
436-
return personList;
437-
}
438-
```
439-
440-
## FAQ ##
441-
442-
### How can I write Primitive Types (``boolean``, ``float``, ``double``)? ###
443-
444-
By default methods like ``mapBoolean`` map the boxed type ``Boolean``, ``Integer``, ``Long``. This might be problematic
445-
if you need to squeeze out the last seconds when doing bulk inserts, see Issue:
446-
447-
* [https://github.com/PgBulkInsert/PgBulkInsert/issues/93](https://github.com/PgBulkInsert/PgBulkInsert/issues/93)
448-
449-
So for every data type that also has a primitive type, you can add a "Primitive" suffix to the method name like:
450-
451-
* ```mapBooleanPrimitive``
452-
453-
This will use the primitive type and prevent boxing and unboxing of values.
454-
455-
### How can I write a ``java.sql.Timestamp``? ###
456-
457-
You probably have Java classes with a ``java.sql.Timestamp`` in your application. Now if you use the ``AbstractMapping`` or a ``SimpleRowWriter`` it expects a ``LocalDateTime``. Here is how to map a ``java.sql.Timestamp``.
458-
459-
Imagine you have an ``EMail`` class with a property ``emailCreateTime``, that is using a ``java.sql.Timestamp`` to
460-
represent the time. The column name in Postgres is ``email_create_time`` and you are using a ``timestamp`` data type.
461-
462-
To map the ``java.sql.Timestamp`` you would write the ``mapTimeStamp`` method like this:
463-
464-
```java
465-
mapTimeStamp("email_create_time", x -> x.getEmailCreateTime() != null ? x.getEmailCreateTime().toLocalDateTime() : null);
466-
```
467-
468-
And here is the complete example:
469-
470-
```java
471-
public class EMail {
472-
473-
private Timestamp emailCreateTime;
474-
475-
public Timestamp getEmailCreateTime() {
476-
return emailCreateTime;
477-
}
478-
}
479-
480-
public static class EMailMapping extends AbstractMapping<EMail>
481-
{
482-
public EMailMapping(String schema) {
483-
super(schema, "unit_test");
484-
485-
mapTimeStamp("email_create_time", x -> x.getEmailCreateTime() != null ? x.getEmailCreateTime().toLocalDateTime() : null);
486-
}
487-
}
488-
```
489-
490-
### Handling Null Characters or... 'invalid byte sequence for encoding "UTF8": 0x00' ###
491-
492-
If you see the error message ``invalid byte sequence for encoding "UTF8": 0x00`` your data contains Null Characters. Although ``0x00`` is totally valid UTF-8... PostgreSQL does not support writing it, because it uses C-style string termination internally.
493-
494-
PgBulkInsert allows you to enable a Null Value handling, that removes all ``0x00`` occurences and replaces them with an empty string:
495-
496-
```java
497-
// Create the Table Definition:
498-
SimpleRowWriter.Table table = new SimpleRowWriter.Table(schema, tableName, columnNames);
499-
500-
// Create the Writer:
501-
SimpleRowWriter writer = new SimpleRowWriter(table);
502-
503-
// Enable the Null Character Handler:
504-
writer.enableNullCharacterHandler();
505-
```
506-
507-
## Running the Tests ##
508-
509-
Running the Tests requires a PostgreSQL database.
510-
511-
You have to configure the test database connection in the module ``pgbulkinsert-core`` and file ``db.properties``:
512-
513-
```ini
514-
db.url=jdbc:postgresql://127.0.0.1:5432/sampledb
515-
db.user=philipp
516-
db.password=test_pwd
517-
db.schema=public
518-
```
519-
520-
The tests are transactional, that means any test data will be rolled back once a test finishes. But it probably makes
521-
sense to set up a separate ``db.schema`` for your tests, if you want to avoid polluting the ``public`` schema or have
522-
different permissions.
523-
524-
## License ##
525-
526-
PgBulkInsert is released with under terms of the [MIT License]:
527-
528-
* [https://github.com/bytefish/PgBulkInsert](https://github.com/bytefish/PgBulkInsert)
529-
530-
531-
## Resources ##
532-
533-
* [Npgsql](https://github.com/npgsql/npgsql)
534-
* [Postgres on the wire - A look at the PostgreSQL wire protocol (PGCon 2014)](https://www.pgcon.org/2014/schedule/attachments/330_postgres-for-the-wire.pdf)
535-
536-

0 commit comments

Comments
 (0)