Stream processing is consuming an input stream or sequence of bytes with some format, processing the data as it arrives, and translating it into another format in a related output stream.
Questions tagged [stream-processing]
89 questions
59
votes
3 answers
Is it an antipattern to use peek() to modify a stream element?
Suppose I have a stream of Things and I want to "enrich" them mid stream, I can use peek() to do this, eg:
streamOfThings.peek(this::thingMutator).forEach(this::someConsumer);
Assume that mutating the Things at this point in the code is correct…
Bohemian
- 2,066
50
votes
4 answers
Why should I use "functional operations" instead of a for loop?
for (Canvas canvas : list) {
}
NetBeans suggests me to use "functional operations":
list.stream().forEach((canvas) -> {
});
But why is this preferred? If anything, it is harder to read and understand. You are calling stream(), then forEach() using…
Saturn
- 3,937
42
votes
4 answers
What is a byte stream actually?
Can anyone explain me what byte stream actually contains? Does it contain bytes (hex data) or binary data or english letters only? I am also confused about the term "raw data". If someone asked me to "reverse the 4 byte data", then what should I…
user2720323
- 991
- 3
- 12
- 11
21
votes
3 answers
Is it a sane thing to return Streams wherever we would normally return Collections?
While developing my API that is not tied to any legacy code, I often find myself writing methods that are purely Streams pipeline terminated by collecting the results. Like this one:
ImmutableSet deriveSomethingMeaningfulFromPrivateState() {
…
jojman
- 313
- 1
- 7
17
votes
2 answers
Traditional Message Brokers and Streaming Data
According to the Kafka site:
"Kakfa is used for building real-time data pipelines and streaming apps."
Searching the internet far and wide, I've found the following generally-accepted definition of what "stream data" is:
Stream data is data that…
smeeb
- 4,950
- 10
- 33
- 52
14
votes
2 answers
Know file size with a base64 string
I do not have very clear what a base64 string is or how it is related to a file.
Right now I have a website that uploads an image as a base64 string to my server. This string is converted back into a bitmap and resized.
However, I would like the…
luisgepeto
- 257
13
votes
1 answer
How does sorting with java 8 stream work under the hood?
When I call Stream.sort(..) is there a new array of elements created and the stream iterates over the newly created sorted array?
In other words, how Java 8 Stream does sort under the hood?
InformedA
- 3,031
10
votes
3 answers
Why do you want to avoid flushing stdout?
I stumbled upon a question in Codereview, and in one answer the feedback was to avoid std::endl because it flushes the stream. The full quote is:
I'd advise avoiding std::endl in general. Along with writing a new-line to the stream, it flushes the…
klutt
- 1,438
8
votes
1 answer
What type of buffer should I implement for a one-way streaming audio device?
I'm working on a project where audio data is streamed to a device. The audio data is encoded via opus and streamed at 20 ms payloads at a time. The streaming is done via TCP to avoid packet loss completely. The goal of the streaming is to have as…
6
votes
1 answer
Download file stream as it's being generated
Scenario
We are building a UI that allows users to query our data in bulk. The return format is CSV and there is a decent amount of processing that needs to happen, so this processing processing is done in parallel and then sets of rows are streamed…
Jared Goguen
- 250
6
votes
2 answers
Are there any techniques for detecting redundancies a stream of changes to a filesystem?
I'm working on a file-synchronization client that currently produces a stream of changes to the underlying filesystem. That is a stream of create/update/move/delete events is produced for each synchronization "target".
Each event includes a…
Louis Thibault
- 2,208
6
votes
1 answer
Why does the C stdio 'ungetc' function exist?
In the C programming language (and many subsequent languages that either directly interfaced with or built a facsimile of the C's Standard IO functions), there exists a function called ungetc: int ungetc(int char, FILE *stream);. It 'puts back' a…
Qqwy
- 4,907
6
votes
1 answer
How is one supposed to deal with the intermediate buffer of DataReader class?
Summary
I am developing a WAV file format reader under WinRT, and for this I need to read random amounts of structs consisting of fundamental types such as int, uint, float and so on.
Back in desktop development one would rely on BinaryReader, now…
aybe
- 955
5
votes
2 answers
What are the distinction and relation between batch processing and stream processing systems?
Design Data Intensive Applications says
Batch processing systems (offline systems) Chapter 10
A batch processing system takes a large amount of input data, runs a
job to pro‐ cess it, and produces some output data. Jobs often
take a …
Tim
- 5,545
5
votes
2 answers
Differences between streams and iterators in C++?
I'm working on code which will provide a generic C++ interface to relational datasets. Some of the data will be in tables in memory (e.g. std::set or std::vector), but some of it will be computed - either produced by a function computing results of…
Dennis
- 206