Skip to content

Commit e167165

Browse files
MichalFoksaMichalFoksa
authored
logicalType switch. Logical types are not generated by default. (#293)
1 parent 8aacf59 commit e167165

11 files changed

+264
-43
lines changed

‎avro/README.md

Lines changed: 80 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -112,39 +112,97 @@ byte[] avroData = mapper.writer(schema)
112112

113113
and that's about it, for now.
114114

115+
## Avro Logical Types
116+
117+
Following is an extract from [Logical Types](http://avro.apache.org/docs/current/spec.html#Logical+Types) paragraph in
118+
Avro schema specification:
119+
> A logical type is an Avro primitive or complex type with extra attributes to represent a derived type. The attribute
120+
> `logicalType` is always be present for a logical type, and is a string with the name of one of the logical types
121+
> defined by Avro specification.
122+
123+
Generation of logical types for limited set of `java.time` classes is supported at the moment. See a table bellow.
124+
125+
### Mapping to Logical Type
126+
127+
Mapping to Avro type and logical type works in few steps:
128+
1. Serializer for particular Java type (or class) determines a Jackson type where the Java type will be serialized into.
129+
2. `AvroSchemaGenerator` determines corresponding Avro type for that Jackson type.
130+
2. If logical type generation is enabled, then `logicalType` is determined for the above combination of Java type and
131+
Avro type.
132+
133+
#### Java type to Avro Logical Type mapping
134+
135+
| Java type | Serialization type | Generated Avro schema with Avro type and logical type
136+
| ----------------------------- | ------------------ | -----------------------------------------------------
137+
| `java.time.OffsetDateTime` | NumberType.LONG | `{"type": "long", "logicalType": "timestamp-millis"}`
138+
| `java.time.ZonedDateTime` | NumberType.LONG | `{"type": "long", "logicalType": "timestamp-millis"}`
139+
| `java.time.Instant` | NumberType.LONG | `{"type": "long", "logicalType": "timestamp-millis"}`
140+
| `java.time.LocalDate` | NumberType.INT | `{"type": "int", "logicalType": "date"}`
141+
| `java.time.LocalTime` | NumberType.INT | `{"type": "int", "logicalType": "time-millis"}`
142+
| `java.time.LocalDateTime` | NumberType.LONG | `{"type": "long", "logicalType": "local-timestamp-millis"}`
143+
144+
_Provided Avro logical type generation is enabled._
145+
146+
### Usage
147+
148+
Call `AvroSchemaGenerator.enableLogicalTypes()` method to enable Avro schema with logical type generation.
149+
150+
```java
151+
// Create and configure Avro mapper. With for example a module or a serializer.
152+
AvroMapper mapper = AvroMapper.builder()
153+
.build();
154+
155+
AvroSchemaGenerator gen = new AvroSchemaGenerator();
156+
// Enable logical types
157+
gen.enableLogicalTypes();
158+
159+
mapper.acceptJsonFormatVisitor(RootType.class, gen);
160+
Schema actualSchema = gen.getGeneratedSchema().getAvroSchema();
161+
```
162+
163+
_**Note:** For best performance with `java.time` classes configure `AvroMapper` to use `AvroJavaTimeModule`. More on
164+
`AvroJavaTimeModule` bellow._
165+
115166
## Java Time Support
116-
Serialization and deserialization support for limited set of `java.time` classes to Avro with [logical type](http://avro.apache.org/docs/current/spec.html#Logical+Types) is provided by `AvroJavaTimeModule`.
117167

118-
This module is to be used either:
119-
- Instead of Java 8 date/time module (`com.fasterxml.jackson.datatype.jsr310.JavaTimeModule`) or
168+
`AvroJavaTimeModule` is the best companionship to enabled to Avro logical types. It provides serialization and
169+
deserialization for set of `java.time` classes into a simple numerical value, e.g., `OffsetDateTime` to `long`,
170+
`LocalTime` to `int`, etc.
171+
172+
| WARNING: Time zone information is lost at serialization. After deserialization, time instant is reconstructed but not the original time zone.|
173+
| --- |
174+
175+
Because data is serialized into simple numerical value (long or int), time zone information is lost at serialization.
176+
Serialized values represent point in time, independent of a particular time zone or calendar. Upon reading a value back,
177+
time instant is reconstructed but not the original time zone.
178+
179+
`AvroJavaTimeModule` is to be used either as:
180+
- replacement of Java 8 date/time module (`com.fasterxml.jackson.datatype.jsr310.JavaTimeModule`) or
120181
- to override Java 8 date/time module and for that, module must be registered AFTER Java 8 date/time module (last registration wins).
121182

183+
### Java types supported by AvroJavaTimeModule, and their mapping to Jackson types
184+
185+
| Java type | Serialization type
186+
| ----------------------------- | ------------------
187+
| `java.time.OffsetDateTime` | NumberType.LONG
188+
| `java.time.ZonedDateTime` | NumberType.LONG
189+
| `java.time.Instant` | NumberType.LONG
190+
| `java.time.LocalDate` | NumberType.INT
191+
| `java.time.LocalTime` | NumberType.INT
192+
| `java.time.LocalDateTime` | NumberType.LONG
193+
194+
### Usage
195+
122196
```java
123197
AvroMapper mapper = AvroMapper.builder()
124198
.addModule(new AvroJavaTimeModule())
125199
.build();
126200
```
127-
128-
#### Note
129-
Please note that time zone information is lost at serialization. Serialized values represent point in time,
130-
independent of a particular time zone or calendar. Upon reading a value back time instant is reconstructed but not the original time zone.
131-
132-
#### Supported java.time types:
133-
134-
Supported java.time types with Avro schema.
135-
136-
| Type | Avro schema
137-
| ------------------------------ | -------------
138-
| `java.time.OffsetDateTime` | `{"type": "long", "logicalType": "timestamp-millis"}`
139-
| `java.time.ZonedDateTime` | `{"type": "long", "logicalType": "timestamp-millis"}`
140-
| `java.time.Instant` | `{"type": "long", "logicalType": "timestamp-millis"}`
141-
| `java.time.LocalDate` | `{"type": "int", "logicalType": "date"}`
142-
| `java.time.LocalTime` | `{"type": "int", "logicalType": "time-millis"}`
143-
| `java.time.LocalDateTime` | `{"type": "long", "logicalType": "local-timestamp-millis"}`
144201

145-
#### Precision
202+
### Precision
146203

147-
Avro supports milliseconds and microseconds precision for date and time related LogicalTypes, but this module only supports millisecond precision.
204+
Avro supports milliseconds and microseconds precision for date and time related logical types. `AvroJavaTimeModule`
205+
supports millisecond precision only.
148206

149207
## Generating Avro Schema from POJO definition
150208

‎avro/src/main/java/com/fasterxml/jackson/dataformat/avro/jsr310/AvroJavaTimeModule.java

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -21,8 +21,8 @@
2121
/**
2222
* A module that installs a collection of serializers and deserializers for java.time classes.
2323
*
24-
* This module is to be used either:
25-
* - Instead of Java 8 date/time module (com.fasterxml.jackson.datatype.jsr310.JavaTimeModule) or
24+
* AvroJavaTimeModule module is to be used either as:
25+
* - replacement of Java 8 date/time module (com.fasterxml.jackson.datatype.jsr310.JavaTimeModule) or
2626
* - to override Java 8 date/time module and for that, module must be registered AFTER Java 8 date/time module.
2727
*/
2828
public class AvroJavaTimeModule extends SimpleModule {

‎avro/src/main/java/com/fasterxml/jackson/dataformat/avro/jsr310/ser/AvroInstantSerializer.java

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -24,8 +24,8 @@
2424
* Please note that time zone information gets lost in this process. Upon reading a value back, we can only
2525
* reconstruct the instant, but not the original representation.
2626
*
27-
* Note: In combination with {@link com.fasterxml.jackson.dataformat.avro.schema.DateTimeVisitor} it aims to produce
28-
* Avro schema with type long and logicalType timestamp-millis:
27+
* Note: In combination with {@link com.fasterxml.jackson.dataformat.avro.schema.AvroSchemaGenerator#enableLogicalTypes()}
28+
* it aims to produce Avro schema with type long and logicalType timestamp-millis:
2929
* {
3030
* "type" : "long",
3131
* "logicalType" : "timestamp-millis"

‎avro/src/main/java/com/fasterxml/jackson/dataformat/avro/jsr310/ser/AvroLocalDateSerializer.java

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -18,8 +18,8 @@
1818
* Serialized value represents number of days from the unix epoch, 1 January 1970 with no reference
1919
* to a particular time zone or time of day.
2020
*
21-
* Note: In combination with {@link com.fasterxml.jackson.dataformat.avro.schema.DateTimeVisitor} it aims to produce
22-
* Avro schema with type int and logicalType date:
21+
* Note: In combination with {@link com.fasterxml.jackson.dataformat.avro.schema.AvroSchemaGenerator#enableLogicalTypes()}
22+
* it aims to produce Avro schema with type int and logicalType date:
2323
* {
2424
* "type" : "int",
2525
* "logicalType" : "date"

‎avro/src/main/java/com/fasterxml/jackson/dataformat/avro/jsr310/ser/AvroLocalDateTimeSerializer.java

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -20,8 +20,8 @@
2020
* Serialized value represents timestamp in a local timezone, regardless of what specific time zone
2121
* is considered local, with a precision of one millisecond from 1 January 1970 00:00:00.000.
2222
*
23-
* Note: In combination with {@link com.fasterxml.jackson.dataformat.avro.schema.DateTimeVisitor} it aims to produce
24-
* Avro schema with type long and logicalType local-timestamp-millis:
23+
* Note: In combination with {@link com.fasterxml.jackson.dataformat.avro.schema.AvroSchemaGenerator#enableLogicalTypes()}
24+
* it aims to produce Avro schema with type long and logicalType local-timestamp-millis:
2525
* {
2626
* "type" : "long",
2727
* "logicalType" : "local-timestamp-millis"

‎avro/src/main/java/com/fasterxml/jackson/dataformat/avro/jsr310/ser/AvroLocalTimeSerializer.java

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -18,9 +18,9 @@
1818
* Serialized value represents time of day, with no reference to a particular calendar,
1919
* time zone or date, where the int stores the number of milliseconds after midnight, 00:00:00.000.
2020
*
21-
* Note: In combination with {@link com.fasterxml.jackson.dataformat.avro.schema.DateTimeVisitor} it aims to produce
22-
* Avro schema with type int and logicalType time-millis:
23-
* {
21+
* Note: In combination with {@link com.fasterxml.jackson.dataformat.avro.schema.AvroSchemaGenerator#enableLogicalTypes()}
22+
* it aims to produce Avro schema with type int and logicalType time-millis:
23+
* {
2424
* "type" : "int",
2525
* "logicalType" : "time-millis"
2626
* }

‎avro/src/main/java/com/fasterxml/jackson/dataformat/avro/schema/DateTimeVisitor.java

Lines changed: 9 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -39,7 +39,8 @@ public Schema builtAvroSchema() {
3939

4040
Schema schema = AvroSchemaHelper.numericAvroSchema(_type);
4141
if (_hint != null) {
42-
String logicalType = logicalType(_hint);
42+
String logicalType = getLogicalType(schema.getType(), _hint);
43+
4344
if (logicalType != null) {
4445
schema.addProp(LogicalType.LOGICAL_TYPE_PROP, logicalType);
4546
} else {
@@ -49,26 +50,26 @@ public Schema builtAvroSchema() {
4950
return schema;
5051
}
5152

52-
private String logicalType(JavaType hint) {
53+
private String getLogicalType(Schema.Type avroType, JavaType hint) {
5354
Class<?> clazz = hint.getRawClass();
5455

55-
if (OffsetDateTime.class.isAssignableFrom(clazz)) {
56+
if (OffsetDateTime.class.isAssignableFrom(clazz) && Schema.Type.LONG == avroType) {
5657
return TIMESTAMP_MILLIS;
5758
}
58-
if (ZonedDateTime.class.isAssignableFrom(clazz)) {
59+
if (ZonedDateTime.class.isAssignableFrom(clazz) && Schema.Type.LONG == avroType) {
5960
return TIMESTAMP_MILLIS;
6061
}
61-
if (Instant.class.isAssignableFrom(clazz)) {
62+
if (Instant.class.isAssignableFrom(clazz) && Schema.Type.LONG == avroType) {
6263
return TIMESTAMP_MILLIS;
6364
}
6465

65-
if (LocalDate.class.isAssignableFrom(clazz)) {
66+
if (LocalDate.class.isAssignableFrom(clazz) && Schema.Type.INT == avroType) {
6667
return DATE;
6768
}
68-
if (LocalTime.class.isAssignableFrom(clazz)) {
69+
if (LocalTime.class.isAssignableFrom(clazz) && Schema.Type.INT == avroType) {
6970
return TIME_MILLIS;
7071
}
71-
if (LocalDateTime.class.isAssignableFrom(clazz)) {
72+
if (LocalDateTime.class.isAssignableFrom(clazz) && Schema.Type.LONG == avroType) {
7273
return LOCAL_TIMESTAMP_MILLIS;
7374
}
7475

‎avro/src/main/java/com/fasterxml/jackson/dataformat/avro/schema/VisitorFormatWrapperImpl.java

Lines changed: 35 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,8 @@ public class VisitorFormatWrapperImpl
2727

2828
protected final DefinedSchemas _schemas;
2929

30+
protected boolean _logicalTypesEnabled = false;
31+
3032
/**
3133
* Visitor used for resolving actual Schema, if structured type
3234
* (or one with complex configuration)
@@ -49,8 +51,21 @@ public VisitorFormatWrapperImpl(DefinedSchemas schemas, SerializerProvider p) {
4951
_provider = p;
5052
}
5153

54+
55+
protected VisitorFormatWrapperImpl(VisitorFormatWrapperImpl src) {
56+
this._schemas = src._schemas;
57+
this._provider = src._provider;
58+
this._logicalTypesEnabled = src._logicalTypesEnabled;
59+
}
60+
61+
/**
62+
* Creates new {@link VisitorFormatWrapperImpl} instance with shared schemas,
63+
* serialization provider and same configuration.
64+
*
65+
* @return new instance with shared properties and configuration.
66+
*/
5267
protected VisitorFormatWrapperImpl createChildWrapper() {
53-
return new VisitorFormatWrapperImpl(_schemas, _provider);
68+
return new VisitorFormatWrapperImpl(this);
5469
}
5570

5671
@Override
@@ -85,6 +100,24 @@ public Schema getAvroSchema() {
85100
return _builder.builtAvroSchema();
86101
}
87102

103+
/**
104+
* Enables Avro schema with Logical Types generation.
105+
*/
106+
public void enableLogicalTypes() {
107+
_logicalTypesEnabled = true;
108+
}
109+
110+
/**
111+
* Disables Avro schema with Logical Types generation.
112+
*/
113+
public void disableLogicalTypes() {
114+
_logicalTypesEnabled = false;
115+
}
116+
117+
public boolean isLogicalTypesEnabled() {
118+
return _logicalTypesEnabled;
119+
}
120+
88121
/*
89122
/**********************************************************************
90123
/* Callbacks
@@ -166,7 +199,7 @@ public JsonIntegerFormatVisitor expectIntegerFormat(JavaType type) {
166199
return null;
167200
}
168201

169-
if (_isDateTimeType(type)) {
202+
if (isLogicalTypesEnabled() && _isDateTimeType(type)) {
170203
DateTimeVisitor v = new DateTimeVisitor(type);
171204
_builder = v;
172205
return v;

‎avro/src/test/java/com/fasterxml/jackson/dataformat/avro/jsr310/AvroJavaTimeModule_schemaCreationTest.java

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -55,6 +55,7 @@ public void testSchemaCreation() throws JsonMappingException {
5555
.addModule(new AvroJavaTimeModule())
5656
.build();
5757
AvroSchemaGenerator gen = new AvroSchemaGenerator();
58+
gen.enableLogicalTypes();
5859

5960
// WHEN
6061
mapper.acceptJsonFormatVisitor(testClass, gen);

‎avro/src/test/java/com/fasterxml/jackson/dataformat/avro/schema/DateTimeVisitor_builtAvroSchemaTest.java

Lines changed: 100 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,100 @@
1+
package com.fasterxml.jackson.dataformat.avro.schema;
2+
3+
import com.fasterxml.jackson.core.JsonParser;
4+
import com.fasterxml.jackson.databind.type.TypeFactory;
5+
import org.apache.avro.LogicalType;
6+
import org.apache.avro.Schema;
7+
import org.apache.avro.specific.SpecificData;
8+
import org.junit.Test;
9+
import org.junit.runner.RunWith;
10+
import org.junit.runners.Parameterized;
11+
import org.junit.runners.Parameterized.Parameter;
12+
import org.junit.runners.Parameterized.Parameters;
13+
14+
import java.time.Instant;
15+
import java.time.LocalDate;
16+
import java.time.LocalDateTime;
17+
import java.time.LocalTime;
18+
import java.time.OffsetDateTime;
19+
import java.time.ZonedDateTime;
20+
import java.util.Arrays;
21+
import java.util.Collection;
22+
23+
import static org.assertj.core.api.Assertions.assertThat;
24+
25+
@RunWith(Parameterized.class)
26+
public class DateTimeVisitor_builtAvroSchemaTest {
27+
28+
private static final TypeFactory TYPE_FACTORY = TypeFactory.defaultInstance();
29+
30+
@Parameter(0)
31+
public Class testClass;
32+
33+
@Parameter(1)
34+
public JsonParser.NumberType givenNumberType;
35+
36+
@Parameter(2)
37+
public Schema.Type expectedAvroType;
38+
39+
@Parameter(3)
40+
public String expectedLogicalType;
41+
42+
@Parameters(name = "With {0} and number type {1}")
43+
public static Collection testData() {
44+
return Arrays.asList(new Object[][]{
45+
// Java type | given number type, | expected Avro type | expected logicalType
46+
{
47+
Instant.class,
48+
JsonParser.NumberType.LONG,
49+
Schema.Type.LONG,
50+
"timestamp-millis"},
51+
{
52+
OffsetDateTime.class,
53+
JsonParser.NumberType.LONG,
54+
Schema.Type.LONG,
55+
"timestamp-millis"},
56+
{
57+
ZonedDateTime.class,
58+
JsonParser.NumberType.LONG,
59+
Schema.Type.LONG,
60+
"timestamp-millis"},
61+
{
62+
LocalDateTime.class,
63+
JsonParser.NumberType.LONG,
64+
Schema.Type.LONG,
65+
"local-timestamp-millis"},
66+
{
67+
LocalDate.class,
68+
JsonParser.NumberType.INT,
69+
Schema.Type.INT,
70+
"date"},
71+
{
72+
LocalTime.class,
73+
JsonParser.NumberType.INT,
74+
Schema.Type.INT,
75+
"time-millis"},
76+
});
77+
}
78+
79+
@Test
80+
public void builtAvroSchemaTest() {
81+
// GIVEN
82+
DateTimeVisitor dateTimeVisitor = new DateTimeVisitor(TYPE_FACTORY.constructSimpleType(testClass, null));
83+
dateTimeVisitor.numberType(givenNumberType);
84+
85+
// WHEN
86+
Schema actualSchema = dateTimeVisitor.builtAvroSchema();
87+
88+
System.out.println(testClass.getName() + " schema:\n" + actualSchema.toString(true));
89+
90+
// THEN
91+
assertThat(actualSchema.getType()).isEqualTo(expectedAvroType);
92+
assertThat(actualSchema.getProp(LogicalType.LOGICAL_TYPE_PROP)).isEqualTo(expectedLogicalType);
93+
/**
94+
* Having logicalType and java-class is not valid according to
95+
* {@link LogicalType#validate(Schema)}
96+
*/
97+
assertThat(actualSchema.getProp(SpecificData.CLASS_PROP)).isNull();
98+
}
99+
100+
}

0 commit comments

Comments
 (0)