Streaming data: collection, storage, processing, and examples

Streaming data is data that arrives continuously in real time or with minimal delay. Unlike batch processing, where data first accumulates and is then processed, the streaming model lets you react to events as soon as they occur.

Such data is widely used in modern systems: from server and IoT device monitoring to financial transactions and real-time chats.

What is streaming data

Streaming data is a sequence of events arriving over time. Each event can contain:

• timestamp (time)
• payload (data)
• metadata (source, type, etc.)

Example sources:

• IoT sensors
• user clicks on a website
• transactions in payment systems
• chat messages
• application logs

Advantages of stream processing

• Minimal latency (low latency)
• Real-time response
• Scalability
• Suited for big data

Disadvantages and complexities

• Architectural complexity
• Error handling and retries (idempotency)
• Managing event order
• Infrastructure requirements

Collecting streaming data (Ingestion)

Ingestion is the first stage, where data enters the system.

Main approaches:

1. HTTP / REST API
   Clients send events via POST requests.
2. Message Brokers (message queues)
   Specialized systems for accepting and buffering streams:
   • Apache Kafka
   • RabbitMQ
   • Amazon Kinesis
3. WebSockets / Streaming protocols
   Used for two-way real-time interaction (e.g., chats).
4. Log agents and collectors
   • Fluentd
   • Logstash

Storing streaming data

Streaming data is rarely stored “as is” — usually it is:

• buffered
• aggregated
• saved to specialized storage

Storage options:

1. Event queues and logs

• Apache Kafka
  Stores the stream as an append-only event log

2. Time-series databases

• InfluxDB
• Prometheus

Ideal for metrics and monitoring

3. NoSQL databases

• MongoDB
• Cassandra

Suited for large volumes and horizontal scaling

4. Data Lake / storage

• Amazon S3
• Hadoop

Used for long-term storage

Processing streaming data

Processing is the key stage where data is turned into useful information.

Main processing types:

1. Real-time processing

Data is processed immediately upon arrival
Examples:

• notifications
• anti-fraud systems
• chat messages

Tools:

• Apache Flink
• Apache Spark Streaming

2. Window-based processing

Processing data over “time windows”:

• sliding window
• tumbling window

Example: average temperature over the last 5 minutes

3. Event-driven architecture

The system reacts to events:

• user places an order → notification trigger
• payment goes through → status update

4. Complex Event Processing (CEP)

Analysis of complex events:

• sequences of actions
• event correlation

Features of processing streaming audio and video data

Processing streaming audio and video differs greatly from ordinary streaming data (logs, events). Here requirements are added for timing, quality, and playback continuity.

When video and audio count as streaming data

Video/audio is considered streaming if it:

• is transmitted as a continuous stream
• plays back as it loads, rather than after fully loading
• is split into small chunks (segments)
• has a time-based nature (time-based data)

Examples:

• a YouTube stream
• online radio on Spotify
• video calls (Zoom, WebRTC)
• IP cameras (RTSP stream)

Here the data flows as a stream of events:

[chunk1] → [chunk2] → [chunk3] → ...

When video and audio are NOT streaming data

If a video or audio file:

• is downloaded in full (e.g., an MP4 file)
• is stored as a file and read in its entirety

Then it is ordinary (batch) data, not streaming.

Types of streaming media

1. Live streaming (real time)

• broadcasts
• webinars
• game streaming

Key feature: minimal latency

2. On-demand streaming

• Netflix, YouTube
• the user chooses what to watch

Key features:

• buffering is present
• seeking is possible

How this relates to classic data streaming

A video/audio stream is a special case of streaming data:

A video/audio stream is usually:

• encoded (H.264, AAC)
• split into segments (HLS, DASH)
• transmitted over HTTP/WebRTC

Yes, video and audio are streaming data if:

• they are transmitted and processed in real time or in parts

But:

• as soon as they become a complete file — it is no longer streaming, but ordinary data

Figure: Architecture of streaming video processing

Let's break down the key features

1. Time sensitivity (real-time constraints)

For media, when the data arrived matters, not just what arrived.

• latency is critical
• data quickly becomes “stale”
• a dropped frame is better than a delay

Example:

• video call → a delay > 300 ms is already noticeable
• a stream → buffering of 2–10 seconds is acceptable

Unlike logs, which can be processed later, this requires near-instant processing

2. Processing by time frames

Video and audio are split into:

• frames
• chunks (segments)

Processing happens not by events, but by a continuous time sequence.

Example:

frame1 → frame2 → frame3 → ...

Important:

• order is strictly required
• events cannot be “shuffled”

3. Encoding and decoding

Before processing, data goes through:

• encoding (encoder)
• decoding (decoder)

Popular formats:

• video: H.264, H.265
• audio: AAC, Opus

This adds:

• CPU/GPU load
• latency (encoding latency)

4. Buffering

To avoid lags, a buffer is used:

• the client preloads several seconds of data in advance
• network hiccups are compensated for by the buffer

Balance:

• large buffer → stability, but latency
• small buffer → low latency, but risk of lags

5. Adaptive Bitrate Streaming

Stream quality changes in real time:

• poor internet → low quality
• good internet → HD / 4K

Protocols:

• HLS
• MPEG-DASH

The system dynamically switches the stream

6. Data loss tolerance (lossy tolerance)

Unlike business data:

• losing some packets is acceptable
• the main thing is to preserve smoothness

Example:

• 1 frame is dropped → the user will barely notice
• a 2-second delay → will be noticed immediately

7. Processing “on the fly” (in-stream processing)

Media can be processed directly in the stream:

• transcoding (format conversion)
• compression
• ad insertion
• subtitles
• speech recognition (speech-to-text)

8. Stream synchronization (audio + video sync)

Important:

• audio and video must stay in sync

Issues:

• desync → poor UX
• timestamps must be taken into account

9. Scaling (CDN and edge)

Video streaming almost always uses:

• CDN (Content Delivery Network)
• edge servers

Example:

• Cloudflare
• Akamai

This reduces latency and load

10. Transmission protocols

Different scenarios → different protocols:

11. Stateful processing

Unlike ordinary events:

• state must be maintained:
  • buffer
  • current bitrate
  • playback position

12. Resource constraints

Media streaming:

• requires a lot of bandwidth
• loads the CPU/GPU
• requires optimization

Summary comparison

Processing streaming audio and video is:

a balance between latency, quality, and stability

Unlike classic data streams:

• time continuity matters here
• data loss is acceptable
• complex infrastructure is used (codecs, CDN, buffers)

Use cases

1. Financial systems

Payment systems (e.g., transaction processing):

• real-time fraud detection
• payment confirmation

2. IoT (Internet of Things)

Sensors transmit data:

• temperature
• pressure
• GPS

Applications:

• smart homes
• industry

3. Web applications and analytics

• click tracking
• user behavior
• A/B testing

4. Chats and real-time systems

Related to WebSocket architecture:

• instant message delivery
• UI updates

5. Monitoring and DevOps

• application logs
• CPU, RAM metrics
• alerts

6. Online audio and video broadcasts

Conclusion

Streaming data is the foundation of modern real-time systems. It allows companies to react to events faster, improve user experience, and make decisions based on up-to-date information.

As data volumes grow and processing speed requirements increase, streaming becomes not just an option but a necessity for most high-load systems.

Streaming data is a continuous flow of events that:

• is collected via APIs and message brokers
• is stored in specialized systems
• is processed in real time

It underpins:

• financial systems
• IoT
• analytics
• real-time applications