How I wired Kafka into a Spring Boot microservice — configuration, serialization, error handling, retries, dead letter topics, and the pitfalls that bit me in production.
Kafka looks simple until you hit a serialization error at 3am with 10,000 messages stuck in a queue. Here's the production setup I use that makes Kafka robust and observable.
1@Configuration
2public class KafkaProducerConfig {
3
4 @Bean
5 public ProducerFactory<String, Object> producerFactory() {
6 Map<String, Object> props = new HashMap<>();
7 props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
8 props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
9 props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, JsonSerializer.class);
10 // Durability settings
11 props.put(ProducerConfig.ACKS_CONFIG, "all"); // wait for all replicas
12 props.put(ProducerConfig.RETRIES_CONFIG, 3);
13 props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true); // no duplicates on retry
14 return new DefaultKafkaProducerFactory<>(props);
15 }
16}1@Bean
2public DefaultErrorHandler errorHandler(KafkaTemplate<?, ?> template) {
3 // Retry 3 times with 1s backoff, then send to DLT
4 DeadLetterPublishingRecoverer recoverer =
5 new DeadLetterPublishingRecoverer(template,
6 (rec, ex) -> new TopicPartition(rec.topic() + ".DLT", rec.partition()));
7
8 FixedBackOff backoff = new FixedBackOff(1000L, 3L);
9 return new DefaultErrorHandler(recoverer, backoff);
10}Always configure a Dead Letter Topic. Without it, a single bad message that fails deserialization will block the entire partition indefinitely. Your consumer stops processing all subsequent messages.
For financial or critical operations, you need exactly-once semantics. Use Kafka transactions combined with your DB transaction — the outbox pattern is the most reliable approach.
1@Service
2@Transactional // single DB transaction
3public class OrderService {
4
5 private final OrderRepository orderRepo;
6 private final OutboxRepository outboxRepo; // same DB as order
7
8 public Order placeOrder(OrderRequest req) {
9 Order order = orderRepo.save(new Order(req));
10
11 // Write event to outbox table in SAME transaction
12 outboxRepo.save(OutboxEvent.builder()
13 .aggregateId(order.getId().toString())
14 .eventType("ORDER_PLACED")
15 .payload(toJson(order))
16 .build());
17
18 return order; // DB commit happens here — both rows or neither
19 }
20}
21
22// Separate scheduler reads outbox and publishes to Kafka
23@Scheduled(fixedDelay = 1000)
24public void publishOutboxEvents() {
25 List<OutboxEvent> events = outboxRepo.findUnpublished();
26 for (OutboxEvent e : events) {
27 kafkaTemplate.send("orders", e.getAggregateId(), e.getPayload());
28 outboxRepo.markPublished(e.getId());
29 }
30}More in Backend Engineering