Kafka streams to build many to one connectivity – Part 1/2

Intended Audience

I have tried creating a simple many to one request/response(asynchronous) service application using Kafka messaging and Kafka Streams to demonstrate its capabilities as messaging and streams processing.

This post is intended for audiences

  • familiar with Kafka messaging and Kafka Streams theoretically and wants to visualize it better using an industry example
  • want to understand setting up Kafka development environment on their development machines and write some code to play hands-on (Part 2)

Use Case

A Global bank has a global account service application that provides an interface to fetch the customer’s account information.

Various client applications across the bank require to consume this service to fetch customer details, Ex:- mobile app, UPI app, retail banking platform, corporate banking platform, etc.

Global accounting service has its own request/response message/API contract.

This is a hybrid model where account service is on the latest technologies like API based and uses Kafka infrastructure to communicate where many clients are still monolithic and use conventional messaging channels like IBM MQ.

Following points to consider

  • All applications having connectivity using messaging technologies like IBM MQ
  • Multiple client applications built on different technologies
  • Client applications have their own request format, Example: vendor’s products having their own format of request generation
  • Accounting service application has its own request-response format
  • The need for message format transformation between the client application and accounting service application for request as well as response flow

Kafka messaging and Kafka Streams in action

{“copyRProtection”:false,”sliderItemTitleEnable”:true,”sliderItemTitleFontSize”:17,”sliderSocialShareEnabled”:false,”galleryPreloaderColor”:”rgba(255,255,255,1)”,”galleryBgColor”:”rgba(255,255,255,0)”,”collectionColumns”:1,”collectionThumbRecomendedWidth”:250,”externalLinkFontSize”:12,”captionHAlign”:”bottom”,”collectionthumbHoverTitleFontSize”:11,”collectionthumbHoverTitleTextBgColor”:”rgba(0,0,0,0.1)”,”galleryType”:”pgc_sgb_masonry”,”galleryId”:”3d9d1ab0_1605601616165″,”images”:[{“id”:61,”title”:”Kafka Streams”,”url”:”https://abhinotes.com/wp-content/uploads/2020/11/Kafka-Streams.jpg”,”link”:”https://abhinotes.com/2020/11/17/kafka-messaging-kafka-streams-to-build-many-to-one-connectivity-part-1-2/kafka-streams-2/”,”alt”:””,”description”:”Use Kafka streams to join messages in realtime time and make decisions, enrich response , etc”,”caption”:”Use Kafka streams to join messages in realtime time and make decisions, enrich response , etc”,”mime”:”image/jpeg”,”type”:”image”,”width”:1638,”height”:918,”sizes”:{“thumbnail”:{“height”:150,”width”:150,”url”:”https://abhinotes.com/wp-content/uploads/2020/11/Kafka-Streams-150×150.jpg”,”orientation”:”landscape”},”medium”:{“height”:168,”width”:300,”url”:”https://abhinotes.com/wp-content/uploads/2020/11/Kafka-Streams-300×168.jpg”,”orientation”:”landscape”},”large”:{“height”:348,”width”:620,”url”:”https://abhinotes.com/wp-content/uploads/2020/11/Kafka-Streams-1024×574.jpg”,”orientation”:”landscape”},”full”:{“url”:”https://abhinotes.com/wp-content/uploads/2020/11/Kafka-Streams.jpg”,”height”:918,”width”:1638,”orientation”:”landscape”}},”meta”:false}],”itemsMetaDataCollection”:[],”galleryData”:”{\u0022copyRProtection\u0022:false,\u0022copyRAlert\u0022:\u0022Hello, this photo is mine!\u0022,\u0022sliderScrollNavi\u0022:false,\u0022sliderNextPrevAnimation\u0022:\u0022animation\u0022,\u0022galleryScrollPositionControll\u0022:false,\u0022sliderItemCounterEnable\u0022:true,\u0022sliderCaptionSrc\u0022:\u0022caption\u0022,\u0022sliderItemTitleEnable\u0022:true,\u0022sliderItemTitleFontSize\u0022:17,\u0022sliderItemTitleTextColor\u0022:\u0022rgba(255,255,255,1)\u0022,\u0022itemCounterColor\u0022:\u0022rgba(255,255,255,1)\u0022,\u0022sliderThumbBarEnable\u0022:true,\u0022sliderThumbBarHoverColor\u0022:\u0022rgba(240,240,240,1)\u0022,\u0022sliderBgColor\u0022:\u0022rgba(0,0,0,0.8)\u0022,\u0022sliderPreloaderColor\u0022:\u0022rgba(240,240,240,1)\u0022,\u0022sliderHeaderFooterBgColor\u0022:\u0022rgba(0,0,0,0.4)\u0022,\u0022sliderNavigationColor\u0022:\u0022rgba(0,0,0,1)\u0022,\u0022sliderNavigationColorOver\u0022:\u0022rgba(255,255,255,1)\u0022,\u0022sliderNavigationIconColor\u0022:\u0022rgba(255,255,255,1)\u0022,\u0022sliderNavigationIconColorOver\u0022:\u0022rgba(0,0,0,1)\u0022,\u0022sliderSlideshow\u0022:true,\u0022sliderSlideshowDelay\u0022:8,\u0022slideshowIndicatorColor\u0022:\u0022rgba(255,255,255,1)\u0022,\u0022slideshowIndicatorColorBg\u0022:\u0022rgba(255,255,255,0.5)\u0022,\u0022sliderThumbSubMenuBackgroundColor\u0022:\u0022rgba(255,255,255,0)\u0022,\u0022sliderThumbSubMenuBackgroundColorOver\u0022:\u0022rgba(255,255,255,1)\u0022,\u0022sliderThumbSubMenuIconColor\u0022:\u0022rgba(255,255,255,1)\u0022,\u0022sliderThumbSubMenuIconHoverColor\u0022:\u0022rgba(0,0,0,1)\u0022,\u0022sliderSocialShareEnabled\u0022:false,\u0022sliderZoomEnable\u0022:true,\u0022sliderFullscreenEnabled\u0022:true,\u0022modaBgColor\u0022:\u0022rgba(0,0,0,0.8)\u0022,\u0022modalIconColor\u0022:\u0022rgba(255,255,255,1)\u0022,\u0022modalIconColorHover\u0022:\u0022rgba(255,255,255,0.8)\u0022,\u0022shareFacebook\u0022:true,\u0022shareTwitter\u0022:true,\u0022sharePinterest\u0022:true,\u0022sliderItemDownload\u0022:true,\u0022shareCopyLink\u0022:true,\u0022deepLinking\u0022:false,\u0022useLightbox\u0022:true,\u0022lazyLoading\u0022:true,\u0022contentNoPadding\u0022:false,\u0022galleryPreloaderColor\u0022:\u0022rgba(255,255,255,1)\u0022,\u0022galleryBgColor\u0022:\u0022rgba(255,255,255,0)\u0022,\u0022galleryBorderRadius\u0022:0,\u0022collectionColumns\u0022:1,\u0022collectionThumbRecomendedWidth\u0022:250,\u0022thumbSpacing\u0022:5,\u0022externalLink\u0022:true,\u0022externalLinkDefName\u0022:\u0022Read More\u0022,\u0022externalLinkTextColor\u0022:\u0022rgba(0,0,0,1)\u0022,\u0022externalLinkBgColor\u0022:\u0022rgba(255,255,255,1)\u0022,\u0022externalLinkFontSize\u0022:12,\u0022captions\u0022:true,\u0022captionSrc\u0022:\u0022caption\u0022,\u0022captionHAlign\u0022:\u0022bottom\u0022,\u0022collectionthumbHoverTitleFontSize\u0022:11,\u0022collectionthumbRoundedCorners\u0022:0,\u0022collectionthumbHoverTitleTextColor\u0022:\u0022rgba(255,255,255,1)\u0022,\u0022collectionthumbHoverTitleTextBgColor\u0022:\u0022rgba(0,0,0,0.1)\u0022,\u0022collectionthumbHoverBgColor\u0022:\u0022rgba(0,0,0,0.4)\u0022,\u0022collectionThumbSubMenuDownload\u0022:true,\u0022collectionThumbSubMenuShare\u0022:true,\u0022collectionThumbSubMenuBgColor\u0022:\u0022rgba(0,0,0,0.4)\u0022,\u0022collectionThumbSubMenuBgColorHover\u0022:\u0022rgba(255,255,255,1)\u0022,\u0022collectionThumbSubMenuIconColor\u0022:\u0022rgba(255,255,255,1)\u0022,\u0022collectionThumbSubMenuIconColorHover\u0022:\u0022rgba(0,0,0,1)\u0022,\u0022galleryType\u0022:\u0022pgc_sgb_masonry\u0022,\u0022galleryId\u0022:\u00223d9d1ab0_1605601616165\u0022,\u0022images\u0022:[{\u0022id\u0022:61,\u0022title\u0022:\u0022Kafka Streams\u0022,\u0022url\u0022:\u0022https://abhinotes.com/wp-content/uploads/2020/11/Kafka-Streams.jpg\u0022,\u0022link\u0022:\u0022https://abhinotes.com/2020/11/17/kafka-messaging-kafka-streams-to-build-many-to-one-connectivity-part-1-2/kafka-streams-2/\u0022,\u0022alt\u0022:\u0022\u0022,\u0022description\u0022:\u0022Use Kafka streams to join messages in realtime time and make decisions, enrich response , etc\u0022,\u0022caption\u0022:\u0022Use Kafka streams to join messages in realtime time and make decisions, enrich response , etc\u0022,\u0022mime\u0022:\u0022image/jpeg\u0022,\u0022type\u0022:\u0022image\u0022,\u0022width\u0022:1638,\u0022height\u0022:918,\u0022sizes\u0022:{\u0022thumbnail\u0022:{\u0022height\u0022:150,\u0022width\u0022:150,\u0022url\u0022:\u0022https://abhinotes.com/wp-content/uploads/2020/11/Kafka-Streams-150×150.jpg\u0022,\u0022orientation\u0022:\u0022landscape\u0022},\u0022medium\u0022:{\u0022height\u0022:168,\u0022width\u0022:300,\u0022url\u0022:\u0022https://abhinotes.com/wp-content/uploads/2020/11/Kafka-Streams-300×168.jpg\u0022,\u0022orientation\u0022:\u0022landscape\u0022},\u0022large\u0022:{\u0022height\u0022:348,\u0022width\u0022:620,\u0022url\u0022:\u0022https://abhinotes.com/wp-content/uploads/2020/11/Kafka-Streams-1024×574.jpg\u0022,\u0022orientation\u0022:\u0022landscape\u0022},\u0022full\u0022:{\u0022url\u0022:\u0022https://abhinotes.com/wp-content/uploads/2020/11/Kafka-Streams.jpg\u0022,\u0022height\u0022:918,\u0022width\u0022:1638,\u0022orientation\u0022:\u0022landscape\u0022}},\u0022meta\u0022:false}],\u0022itemsMetaDataCollection\u0022:[],\u0022useClobalSettings\u0022:false,\u0022orderBy\u0022:\u0022custom\u0022}”}(function(){if(window.PGC_SGB && window.PGC_SGB.searcher){window.PGC_SGB.searcher.initBlocks()}})()

You may also like...

Leave a Reply

Your email address will not be published. Required fields are marked *