AsyncAPI Logo
On this page

Adeo Group

Industry: RetailCustomers: 500MRevenue: 25.6B EURO turnover, including 768M EURO online.

Adeo owns different brands in retail industry focused on home improvement and DIY markets, like Leroy Merlin.

tl;dr just go and have a look atfull production-used AsyncAPI document

Adeo Group

Challenges

Cost Component Repository product, part of the ADEO tech products, is used to calculate and publish transfer prices between different internal locations globally. Different business units use different information systems. It is hard to learn how each business unit shares information about its systems, API and accuracy.

The initial solution was a developer portal with a list of all applications and reference to dedicated documentation. Some legacy systems had docs written in MS Excel.

There was a need for a standart way of describing event-driven architecture.

Solution

The API is now described with AsyncAPI. The AsyncAPI file, stored with the source code, generates HTML documentation in the same release pipeline for the product. Documentation is exposed internally as part of the product for other company units depending on the API.

Payloads are described with Avro schema. These schemas generate models and are referenced directly in AsyncAPI files thanks to the schemaFormat feature and $ref. This way, they make sure the code is aligned with the docs.

Shift to using AsyncAPI also enables the team to implement more use cases using AsyncAPI files.

Use Case

Document the API of the product, so its users know how it works and how to use it. AsyncAPI was selected as the standard that allows you to generate documentation from a machine-readable document that describes the API. The goal was to document API in a standardized way, so other internal products could follow to unify how APIs are documented across the company.

More Details

Languages: JavaFrameworks: SpringProtocols: Kafka

Testing strategy

For Kafka, e2e tests are done with Zerocode. Load tests are handled with JMeter with the kloadgen plugin that supports Kafka and Avro.

Approach to code generation

Java models generation. Avro schemas used as a source.

Architecture

The following enterprise integration patterns are applied:

  • Request/Reply Described with description field in AsyncAPI. Reply goes to dedicated reply channel. Example description of response channel: 
    1description: >
    2This topic is used to REPLY Costing Requests and is targeted by the
    3`REPLY_TOPIC` header.
  • Return Address Info that needs to be provided by the client so producer knows where to send a response. Information is sent in the message header with the REPLY_TOPIC property. The AsyncAPI file documents information as part of the Message Header object. Example of request message header with REPLY_TOPIC: 
    1headers:
    2  type: object
    3  required:
    4    - REPLY_TOPIC
    5  properties:
    6    REPLY_TOPIC:
    7      $ref: "#/components/schemas/ReplyTopic"
  • Correlation Identifier This pattern enables the identification of the request given to the sent response. The REQUEST_ID property is in the request message header. The CORRELATION_ID property is in the response message header. Both headers are described in the AsyncAPI Message Header object and referred to in the AsyncAPI correlationID property. This means that correlation identifier is represented by different property in the message header, depending if it is a request or reply. Example of request message header with REQUEST_ID: 
    1headers:
    2  type: object
    3  required:
    4    - REQUEST_ID
    5  properties:
    6    REQUEST_ID:
    7      $ref: "#/components/schemas/RequestId"
    Example of how correlationId points to REQUEST_ID: 
    1correlationId:
    2  description: >
    3    This correlation ID is used for message tracing and messages
    4    correlation.
    5    This correlation ID is generated at runtime based on the `REQUEST_ID`
    6    and sent to the RESPONSE message.
    7  location: $message.header#/REQUEST_ID
  • DeadLetter Channel Also known as Dead Letter Queue. In Kafka, it is just another channel where undelivered messages are sent. Not part of the AsyncAPI file, as API consumers will not listen to this channel. Consumers know what happens with wrong events.
  • Invalid Message Channel Invalid messages are routed to the dedicated channel for rejected requests but are not part of the AsyncAPI file, as API consumers will not listen to this channel. Consumers know what happens with wrong events.

Architecture Diagram

More Details about AsyncAPI

Version: 3.0.0Who maintains documents: DevelopersInternal users: trueExternal users: false

How AsyncAPI documents are stored

Git repository where source code is.

Where maintainers edit AsyncAPI documents

IntelliJ without any special plugins. Sometimes people use AsyncAPI Studio, but not regularly because of lack of support for references to local drive.

What extensions are used

Extensions are used to describe details about custom security:

1  x-sasl.jaas.config: >-
2    org.apache.kafka.common.security.plain.PlainLoginModule required
3    username="<CLUSTER_API_KEY>" password="<CLUSTER_API_SECRET>";
4  x-security.protocol: SASL_SSL
5  x-ssl.endpoint.identification.algorithm: https
6  x-sasl.mechanism: PLAIN

How documentation is generated

Documentation generated from AsyncAPI is hosted as part of the product on a dedicated endpoint using Spring controller. Publishing is part of the CI/CD pipeline for the product using GithubActions.

Related Maven configuration used to trigger docs generation with AsyncAPI Generator industry:

1 <profile>
2        <id>generate-asyncapi-doc</id>
3        <build>
4            <plugins>
5                <plugin>
6                    <groupId>com.github.eirslett</groupId>
7                    <artifactId>frontend-maven-plugin</artifactId>
8                    <!-- Use the latest released version:
9                    https://repo1.maven.org/maven2/com/github/eirslett/frontend-maven-plugin/ -->
10                    <version>${frontend-maven-plugin.version}</version>
11                    <configuration>
12                        <nodeVersion>v12.18.4</nodeVersion>
13                        <installDirectory>${node.installation.path}</installDirectory>
14                        <workingDirectory>${node.installation.path}</workingDirectory>
15                    </configuration>
16                    <executions>
17                        <execution>
18                            <id>install node and npm</id>
19                            <goals>
20                                <goal>install-node-and-npm</goal>
21                            </goals>
22                            <phase>generate-resources</phase>
23                        </execution>
24                        <execution>
25                            <id>install @asyncapi/generator globally</id>
26                            <goals>
27                                <goal>npm</goal>
28                            </goals>
29                            <configuration>
30                                <arguments>install @asyncapi/generator@${asyncapi.generator.version}</arguments>
31                            </configuration>
32                        </execution>
33                    </executions>
34                </plugin>
35                <plugin>
36                    <groupId>org.codehaus.mojo</groupId>
37                    <artifactId>exec-maven-plugin</artifactId>
38                    <version>1.6.0</version>
39

40                    <executions>
41                        <execution>
42                            <id>execute-generation</id>
43                            <goals>
44                                <goal>exec</goal>
45                            </goals>
46                            <phase>generate-resources</phase>
47

48                            <configuration>
49                                <!-- Access binary file in node_modules because it doesn't work on windows otherwise. -->
50                                <executable>${node.modules.installation.path}/${ag.binary.name}</executable>
51                                <commandlineArgs>
52                                    ${project.basedir}/src/docs/asyncapi/asyncapi.yaml @asyncapi/html-template@${asyncapi.htmltemplate.version} -p sidebarOrganization=byTags -p
53                                    version=${project.version} -o ${asyncapi.generation.dir}
54                                </commandlineArgs>
55                            </configuration>
56                        </execution>
57                    </executions>
58                </plugin>
59                <plugin>
60                    <groupId>org.apache.maven.plugins</groupId>
61                    <artifactId>maven-resources-plugin</artifactId>
62                    <executions>
63                        <execution>
64                            <id>copy-resources</id>
65                            <!-- here the phase you need -->
66                            <phase>generate-resources</phase>
67                            <goals>
68                                <goal>copy-resources</goal>
69                            </goals>
70                            <configuration>
71                                <outputDirectory>${asyncapi.generation.dir}/assets</outputDirectory>
72                                <resources>
73                                    <resource>
74                                        <directory>src/docs/asyncapi/assets</directory>
75                                        <filtering>true</filtering>
76                                    </resource>
77                                </resources>
78                            </configuration>
79                        </execution>
80                    </executions>
81                </plugin>
82            </plugins>
83        </build>
84    </profile>

Critical features of AsyncAPI related to documentation:

  • use of version parameter in the generator command to display the release version from the Maven pom
  • descriptions that support CommonMark (Markdown) as they allow to put detailed structured descriptions and screenshots inside generated docs
  • examples and validation information. In this case converted from Avro to JSON Schema to show it in documentation and have examples generated
  • Tags for tagging operations to categorize them to make it easier to navigate in documentation UI

What bindings are used

All Kafka bindings are used. Server, channel, operation and message bindings.

Example of server bindings:

1bindings:
2  kafka:
3    schema.registry.url: >-
4      https://schema-registry.prod.url/

Example of channel bindings:

1bindings:
2  kafka:
3    replicas: 3
4    partitions: 3
5    cleanup.policy: delete
6    retention.ms: 7 days

Example of operation bindings:

1bindings:
2  kafka:
3    groupId:
4      type: string
5      description: >
6        The groupId must be prefixed by your `svc` account, deliver by the
7        Adeo Kafka team.
8        This `svc` must have the write access to the topic.
9    value.subject.name.strategy:
10      type: string
11      description: >
12        We use the RecordNameStrategy to infer the messages schema.
13        Use
14        `value.subject.name.strategy=io.confluent.kafka.serializers.subject.RecordNameStrategy`
15        in your producer configuration.

Example of message bindings:

1bindings:
2  kafka:
3    key:
4      $ref: "https://asyncapi.com/resources/casestudies/adeo/CostingResponseKey.avsc"

What tools are used

Schemas

Spec: Avro 1.9

Storage strategy

Git repository where source code is. During release they are published to Confluent Schema Registry.

Schema Registry

Confluent Schema Registry.

Versioning of schemas

Versioning is based on git tags. The schema version pushed to Confluent Schema Registry matches the git tag version of the product. Every schema has a version information that matches with product tag version.

Example Avro schema with version information:

1{
2  "namespace": "com.adeo.casestudy.costingrequest",
3  "type": "record",
4  "name": "CostingRequestPayload",
5  "version": "1.1.0",
6  "fields": [ ... ]
7}

Validation of message schemas

Based on validation using Confluent Schema Registry.

Additional Resources

Watch this video presentation about AsyncAPI case study from Ludovic Dussart, Ineat & Antoine Delequeuche, Adeo.

Production-use AsyncAPI document

1asyncapi: 3.0.0
2info:
3  title: Adeo AsyncAPI Case Study
4  version: "%REPLACED_BY_MAVEN%"
5  description: >
6    This Adeo specification illustrates how ADEO uses AsyncAPI to document some of their exchanges.
7  contact:
8    name: AsyncAPI Community
9    email: info@asyncapi.io
10  tags:
11    - name: costing
12      description: "Costing channels, used by Costing clients."
13servers:
14  production:
15    host: "prod.url:9092"
16    protocol: kafka-secure
17    description: Kafka PRODUCTION cluster
18    security:
19      - $ref: '#/components/securitySchemes/sasl-ssl'
20    bindings:
21      kafka:
22        schemaRegistryUrl: >-
23          https://schema-registry.prod.url/
24  staging:
25    host: "staging.url:9092"
26    protocol: kafka-secure
27    description: Kafka STAGING cluster for `uat` and `preprod` environments
28    security:
29      - $ref: '#/components/securitySchemes/sasl-ssl'
30    bindings:
31      kafka:
32        schemaRegistryUrl: >-
33          https://schema-registry.staging.url/
34  dev:
35    host: "dev.url:9092"
36    protocol: kafka-secure
37    description: Kafka DEV cluster for `dev` and `sit` environments
38    security:
39      - $ref: '#/components/securitySchemes/sasl-ssl'
40    bindings:
41      kafka:
42        schemaRegistryUrl: >-
43          https://schema-registry.dev.url/
44channels:
45  costingRequest:
46    address: "adeo-{env}-case-study-COSTING-REQUEST-{version}"
47    description: >
48      Use this topic to do a Costing Request to Costing product.
49      We use the
50      [**RecordNameStrategy**](https://docs.confluent.io/platform/current/schema-registry/serdes-develop/index.html#subject-name-strategy)
51      to infer the messages schema.
52      You have to define `x-value.subject.name.strategy` to
53      `io.confluent.kafka.serializers.subject.RecordNameStrategy` in your
54      producer to use the schema we manage.
55      The schema below illustrates how Costing Request messages are
56      handled.
57      ![](https://user-images.githubusercontent.com/5501911/188920831-689cec5f-8dc3-460b-8794-0b54ec8b0ac8.png)
58    parameters:
59      env:
60        $ref: "#/components/parameters/Env"
61      version:
62        $ref: "#/components/parameters/Version"
63    bindings:
64      kafka:
65        replicas: 3
66        partitions: 3
67        topicConfiguration:
68          cleanup.policy: [ "delete" ]
69          retention.ms: 604800000
70    messages:
71      costingRequest:
72        $ref: "#/components/messages/costingRequestV1"
73  costingResponse:
74    address: "adeo-{env}-case-study-COSTING-RESPONSE-{version}"
75    description: >
76      This topic is used to REPLY Costing Requests and is targeted by the
77      `REPLY_TOPIC` header.
78      **You must grant PUBLISH access to our `svc-ccr-app` service account.**.
79      We use the
80      [**RecordNameStrategy**](https://docs.confluent.io/platform/current/schema-registry/serdes-develop/index.html#subject-name-strategy)
81      to infer the messages schema.
82      You have to define `key.subject.name.strategy` and
83      `x-value.subject.name.strategy` to
84      `io.confluent.kafka.serializers.subject.RecordNameStrategy` in your
85      consumer.
86      The schema below illustrates how Costing Response messages are
87      handled.
88           ![](https://user-images.githubusercontent.com/5501911/188920831-689cec5f-8dc3-460b-8794-0b54ec8b0ac8.png)
89    parameters:
90      env:
91        $ref: "#/components/parameters/Env"
92      version:
93        $ref: "#/components/parameters/Version"
94    bindings:
95      kafka:
96        x-key.subject.name.strategy:
97          type: string
98          description: >
99            We use the RecordNameStrategy to infer the messages schema.
100            Use
101            `x-key.subject.name.strategy=io.confluent.kafka.serializers.subject.RecordNameStrategy`
102            in your consumer configuration.
103        x-value.subject.name.strategy:
104          type: string
105          description: >
106            We use the RecordNameStrategy to infer the messages schema.
107            Use
108            `x-value.subject.name.strategy=io.confluent.kafka.serializers.subject.RecordNameStrategy`
109            in your consumer configuration.
110    messages:
111      costingResponse:
112        $ref: "#/components/messages/costingResponse"
113operations:
114    requestCosting:
115      action: receive
116      channel: 
117        $ref: '#/channels/costingRequest'
118      reply:
119        channel: 
120          $ref: '#/channels/costingResponse'
121        address:
122          location: '$message.header#/REPLY_TOPIC'
123      summary: |
124        [COSTING] Request one or more Costing calculation for any product
125      description: >
126        You can try a costing request using our [Conduktor producer
127        template](https://conduktor.url)
128      tags:
129        - name: costing
130      bindings:
131        kafka:
132          groupId:
133            type: string
134            description: >
135              The groupId must be prefixed by your `svc` account, deliver by the
136              Adeo Kafka team.
137              This `svc` must have the write access to the topic.
138          x-value.subject.name.strategy:
139            type: string
140            description: >
141              We use the RecordNameStrategy to infer the messages schema.
142              Use
143              `x-value.subject.name.strategy=io.confluent.kafka.serializers.subject.RecordNameStrategy`
144              in your producer configuration.
145    getCostingResponse:
146      action: send
147      channel: 
148        $ref: '#/channels/costingResponse'
149      summary: >
150        [COSTING] Get the costing responses matching an initial Costing
151        Request.
152      bindings:
153        kafka:
154          groupId:
155            type: string
156            description: >
157              The groupId must be prefixed by your `svc` account, deliver by the
158              Adeo Kafka team.
159              This `svc` must have the read access to the topic.
160      tags:
161        - name: costing
162

163components:
164  correlationIds:
165    costingCorrelationId:
166      description: >
167        This correlation ID is used for message tracing and messages
168        correlation.
169        This correlation ID is generated at runtime based on the `REQUEST_ID`
170        and sent to the RESPONSE message.
171      location: $message.header#/REQUEST_ID
172  messages:
173    costingRequestV1:
174      name: CostingRequestV1
175      title: Costing Request V1
176      summary: Costing Request V1 inputs.
177      tags:
178        - name: costing
179      correlationId:
180        $ref: "#/components/correlationIds/costingCorrelationId"
181      headers:
182        type: object
183        required:
184          - REQUESTER_ID
185          - REQUESTER_CODE
186          - REQUEST_ID
187          - REPLY_TOPIC
188        properties:
189          REQUEST_ID:
190            $ref: "#/components/schemas/RequestId"
191          REPLY_TOPIC:
192            $ref: "#/components/schemas/ReplyTopic"
193          REQUESTER_ID:
194            $ref: "#/components/schemas/RequesterId"
195          REQUESTER_CODE:
196            $ref: "#/components/schemas/RequesterCode"
197      payload:
198        schemaFormat: application/vnd.apache.avro;version=1.9.0
199        schema:
200          $ref: "https://www.asyncapi.com/resources/casestudies/adeo/CostingRequestPayload.avsc"
201    costingResponse:
202      name: CostingResponse
203      title: Costing Response
204      summary: Costing Response ouputs.
205      tags:
206        - name: costing
207      description: >
208        Please refer to the `CostingResponseKey.avsc` schema, available on [our
209        github
210        project](https://github.url/).
211      correlationId:
212        $ref: "#/components/correlationIds/costingCorrelationId"
213      headers:
214        type: object
215        properties:
216          CALCULATION_ID:
217            $ref: "#/components/schemas/MessageId"
218          CORRELATION_ID:
219            $ref: "#/components/schemas/CorrelationId"
220          REQUEST_TIMESTAMP:
221            type: string
222            format: date-time
223            description: Timestamp of the costing request
224          CALCULATION_TIMESTAMP:
225            type: string
226            format: date-time
227            description: Technical timestamp for the costing calculation
228      #bindings:
229      #  kafka:
230      #    key:
231      #      $ref: "https://deploy-preview-921--asyncapi-website.netlify.app/resources/casestudies/adeo/CostingResponseKey.avsc"
232      payload:
233        schemaFormat: application/vnd.apache.avro;version=1.9.0
234        schema: 
235          $ref: "https://deploy-preview-921--asyncapi-website.netlify.app/resources/casestudies/adeo/CostingResponsePayload.avsc"
236  schemas:
237    RequesterId:
238      type: string
239      description: The Costing requester service account used to produce costing request.
240      example: svc-ecollect-app
241    RequesterCode:
242      type: string
243      description: >-
244        The Costing requester code (generally the BU Code). The requester code
245        is useful to get the dedicated context (tenant).
246      example: 1
247    MessageId:
248      type: string
249      format: uuid
250      description: A unique Message ID.
251      example: 1fa6ef40-8f47-40a8-8cf6-f8607d0066ef
252    RequestId:
253      type: string
254      format: uuid
255      description: >-
256        A unique Request ID needed to define a `CORRELATION_ID` for exchanges,
257        which will be sent back in the Costing Responses.
258      example: 1fa6ef40-8f47-40a8-8cf6-f8607d0066ef
259    CorrelationId:
260      type: string
261      format: uuid
262      description: >-
263        A unique Correlation ID defined from the `REQUEST_ID` or the
264        `MESSAGE_ID` provided in the Costing Request.
265      example: 1fa6ef40-8f47-40a8-8cf6-f8607d0066ef
266    BuCode:
267      type: string
268      description: The Business Unit code for which data are applicable.
269      example: 1
270    ReplyTopic:
271      type: string
272      description: >
273        The Kafka topic where to send the Costing Response. This is required for
274        the [Return Address EIP
275        pattern](https://www.enterpriseintegrationpatterns.com/patterns/messaging/ReturnAddress.html).
276        **You must grant WRITE access to our `svc-ccr-app` service account.**
277      example: adeo-case-study-COSTING-RESPONSE-V1
278    ErrorStep:
279      type: string
280      description: |
281        The woker that has thrown the error.
282      example: EXPOSE_RESULT
283    ErrorMessage:
284      type: string
285      description: |
286        The error message describing the error.
287      example: Error message
288    ErrorCode:
289      type: string
290      description: |
291        The error code.
292      example: CURRENCY_NOT_FOUND
293  parameters:
294    Env:
295      description: Adeo Kafka Environement for messages publications.
296      enum:
297          - dev
298          - sit
299          - uat1
300          - preprod
301          - prod
302    Version:
303      # no more sschema
304      # V2 - https://v2.asyncapi.com/docs/reference/specification/v2.6.0#parameterObject
305      # V3 - https://www.asyncapi.com/docs/reference/specification/v3.0.0#parameterObject
306      description: the topic version you want to use
307      examples: 
308        - V1
309      default: V1
310  securitySchemes:
311    sasl-ssl:
312      type: plain
313      x-sasl.jaas.config: >-
314        org.apache.kafka.common.security.plain.PlainLoginModule required
315        username="<CLUSTER_API_KEY>" password="<CLUSTER_API_SECRET>";
316      x-security.protocol: SASL_SSL
317      x-ssl.endpoint.identification.algorithm: https
318      x-sasl.mechanism: PLAIN
319      description: >
320        Use [SASL authentication with SSL
321        encryption](https://docs.confluent.io/platform/current/security/security_tutorial.html#configure-clients)
322        to connect to the ADEO Broker.