Questions tagged [io]
Input/Output of data to/from a system. This usually implies file storage or network communication, but may also mean interaction with a user through a user interface.
95 questions
4
votes
2
answers
267
views
How to structuring a read/write submodule in OOP Python
I am developing a python package that needs to be able to read/write from/to multiple formats. E.g. foo format and bar format. I am trying to contain the functions relating to each format in a single ...
- 149
0
votes
2
answers
304
views
What should I consider when determining what `ALLOC_MAX` should be in this type of I/O loop?
For, e.g. determining an amount of memory that is safe to allocate for processing file or device with this type of I/O loop:
HANDLE hFile /* = file open with GENERIC_READ */;
LARGE_INTEGER liSize;
...
- 361
1
vote
1
answer
403
views
What does it mean to be "truly" asynchronous?
Reading the Proactor pattern paper, specifically this part:
I/O Completion Ports in Windows NT: The Windows NT operating system implements the Proactor pattern. Various Asynchronous Operations such ...
- 179
4
votes
10
answers
4k
views
Can multi-threading improve performance of an IO-bound process?
I am trying to understand the difference between CPU Bound vs IO Bound process. ChatGPT suggested that multi-threading/parallel processing can help a CPU bound process; However, I think that having ...
- 199
0
votes
1
answer
92
views
Handling IO operations through a server and building the UI using a separate framework/application
I am trying to build a file editor, and I wanted to build the UI using Flutter. However I wanted to implement IO operations (reading a file, applying changes, etc) in Rust.
The reason I would like to ...
- 115
3
votes
3
answers
944
views
File I/O - How HDD or SSD works with OS file system? [closed]
I've studied how data transfer with secondary storage(HDD or SSD) works.
Would you mind to check that my understanding is correct?
File system
block is basic read/write logical unit which is used in ...
- 133
2
votes
2
answers
262
views
Operating systems - whose responsibility is it to coordinate process I/O requests?
I am reading Tanenbaum's Modern Operating Systems. I want to understand a particular concept regarding processes and blocking system calls, specifically with regards to I/O. I assume threads might ...
- 396
0
votes
2
answers
1k
views
How does a process know the location of another's stdout, stderr, stdin? [closed]
I want to know how standard I/O channels are handled in more depth.
Processes have their own stdout, stderr, stdin channels where they can read/write. As far as I know, these are stored as temporary ...
- 129
1
vote
1
answer
1k
views
Is Epoll/Multiplexing suitable to make network requests instead of "listen to" incoming requests?
I'm studying asynchronous IO, concurrent models for IO and how things works on windows, linux and most used web frameworks.
I'm struggling on understanding why single-threaded event loops like the one ...
0
votes
1
answer
349
views
Is background long running thread a good choice for disk IO?
I am building a storage engine software that allows concurrent data writing, now I have two different choices here:
Method 1. Background Long-Running Thread
Multiple user threads write to their own ...
- 101
7
votes
3
answers
2k
views
Best way to reduce database writes?
I couldn't find a similar enough thread, so i'm making a new one. My question is about reducing database writes. Context is that I have an application which increments a number in a database every ...
- 251
3
votes
5
answers
810
views
Applying function to file line by line or read entirely into structure first?
I've often found myself with the need to develop tools that process large files over a network and perform an operation to every element in that file. An element may be an individual line or an object ...
- 1,047
2
votes
1
answer
244
views
Are streams of binary data considered a form of bit banging?
Are streams of binary data considered a form of bit banging? Does this definition change if the array is buffered? I am referring software which handles binary data on a general purpose CPU; for ...
- 191
0
votes
1
answer
359
views
What Actually Indicates the end of a Socket Input/Output feed?
In the most abstract, platform agnostic way possible, can someone explain what actually determines the end of input/output a socket? Is this something the programming language itself typically handles ...
- 179
0
votes
2
answers
605
views
Can a C# AnonymousPipeServerStream create a non .NET client?
For example, if I want to write a daemon program in C# that uses anonymous pipes to communicate with programs written in another language, is this both possible and feasible?
I ask because I intend ...
- 179
4
votes
1
answer
2k
views
Difference Between AsyncResult and Task in c#
So, in C#, I understand the historical difference between the two vaguely; a Task is a newer concept and the highest level concurrency native C# offers.
AsyncResult is a bit more ambiguous. For ...
- 179
1
vote
1
answer
141
views
How does disk wiping software access my hard disk?
I'm an amateur programmer. I have a lot of interest in the inner workings of operating systems, a subject that I've been reading a lot about.
What I've understood about kernels is that on most ...
- 121
0
votes
1
answer
260
views
How should we approach layered-architectures that need large amounts of continuous, real-time, communication?
Background
In much of the literature I've read online with respect to multi-layered architectures, many people describe how to create the simple application where:
UI presents static model objects to ...
- 165
1
vote
0
answers
483
views
Android Multiple Threads for I/O Operations in Room
I have a small example that i run on a real device with 4+4 cores.
ExecutorService service = Executors.newFixedThreadPool(4);
for(int i =0 ; i<1000; i++){
Runnable task = new Runnable() {
...
- 117
4
votes
2
answers
5k
views
Sequential or parallel access to hard disk in multithread program?
In multi threaded java program, I initiate four concurrent threads with below details
1. Thread 1 writing file F1
2. Thread 2 writing file F2
3. Thread 3 reading file F3
4. Thread 4 reading ...
- 1,896
2
votes
2
answers
1k
views
Reading huge (up to 32 GB) file in multithreading env?
Goal:
1) huge file read in chunks each 1 MB long, 2) each one gets compressed and written to an another output file.
Note: I am limited to .Net 3.5 only.
Is there a known pattern how to parallelize ...
- 345
-7
votes
1
answer
244
views
What is faster, to read 100MB from file or to compile 100MB of code?
I want to generate a test scene for a unit-test (big file parser). What will be faster, to have a test file and to perform I/O on it, or to generate a large memory buffer using some static parameters ...
- 243
14
votes
6
answers
9k
views
How to Unit-Test a parser of a file?
I'm implementing a metadata parser of image files from all formats. I want to write tests for it. One trivial way to do so is to have test image files of all formats as a resources for the tests, and ...
- 243
7
votes
2
answers
1k
views
What is constraining cross-platform asynchronous file I/O?
Looking at a range of cross-platform languages, libraries and GUI toolkits, I often notice a conspicuous absence of support for asynchronous file I/O. This seems like too much of a common factor to be ...
- 1,615
2
votes
1
answer
580
views
How do peripherals decide which address to use in memory-mapped I/O?
I've been reading about how memory-mapped I/O actually works and I cannot understand how the system assigns an address or address space to a specific device.
Let's say I had two devices that I wanted ...
- 135
1
vote
2
answers
9k
views
One thread for all database operations or one thread for each db operations?
I am working on a server application that should handle many requests. There is one thread per each request and each request has a Database operation.
Creating connection for each request and doing ...
- 113
7
votes
3
answers
8k
views
What does "address space" means when talking about IO devices?
The following quote is from this page:
While some CPU manufacturers implement a single address space in their
chips, others decided that peripheral devices are different from
memory and, ...
- 2,049
11
votes
1
answer
16k
views
How does the Base Address Registers (BARs) in a PCI card work?
I am trying to understand how the Base Address Registers (BARs) in a PCI card work, this is how I think they work:
Each function in a PCI card have 6 BAR fields, and each BAR field is
32-bit in size.
...
- 2,049
2
votes
1
answer
296
views
What does "data bus control" mean?
This video mentions the following:
What does it mean for the DMA controller to be granted the data bus control, does that mean the CPU cannot use the bus to access memory and IO devices until the DMA ...
- 2,049
1
vote
3
answers
1k
views
Can the CPU manipulate the pins of an IO port directly?
Based on what I know so far, when you plug an IO device into an IO port (for example, when you plug a printer into a parallel port), the printer will be represented to the CPU as just another RAM chip....
- 247
4
votes
1
answer
6k
views
How data is accessed in Memory-Mapped I/O?
This is an example of Memory-Mapped I/O:
So basically you access the device controller registers through memory.
Now my question is, when you for example write to the memory location that maps to the ...
- 247
1
vote
1
answer
341
views
Can an IO device have some memory space or can it only have registers?
I am learning about IO devices, and so far I have only seen examples of IO devices that have registers and no memory space. For example, this is a printer that have three registers and no memory space:...
- 247
3
votes
3
answers
8k
views
Are "Control register" and "Status register" and "Data register" part of the device itself?
I am studying about Memory-Mapped I/O from here. I have read the following:
From the CPU's perspective, an I/O device appears as a set of
special-purpose registers, of three general types:
...
- 247
5
votes
3
answers
2k
views
Providing a non-blocking IO API in a C library
I am working on a C library (SlipRock) for interprocess communication.
The library currently exposes a simple, blocking API. This is easy to use, makes misuse (relatively) difficult (this is C after ...
- 826
10
votes
2
answers
10k
views
Separate Thread Pools for I/O and CPU Tasks
I've been puzzling over a good implementation for this for a while. I have a program that does a long-running I/O operation (downloading a file) and a long-running CPU operation (parsing its contents)...
- 229
1
vote
0
answers
191
views
Concurrently parsing records in a binary file in Go
I have a binary file that I want to parse. The file is broken up into records that are 1024 bytes each. The high level steps needed are:
Read 1024 bytes at a time from the file.
Parse each 1024-byte "...
- 127
12
votes
7
answers
2k
views
How do I apply TDD to read/write functions?
It seems like a chicken and egg problem.
You can make a write function write to some data store, but never know you saved it properly without a tested read function.
You can make a read function ...
- 945
0
votes
1
answer
245
views
How are low-level I/O models mapped into stream I/O abstraction?
High-level languages often expose stream-based I/O abstraction to the programmer, where blocking or non-blocking streams offer select/read/write operations. (AFAIK, message-based I/O seems is usually ...
- 1,115
3
votes
4
answers
749
views
Does a full duplex communication object conflict with single responsibility?
I'm writing a Websocket implementation in Java, and I've set it up so that it basically just wraps an InputStream and OutputStream, and has public methods for both sending and receiving. While input ...
- 1,764
3
votes
1
answer
281
views
What are good conventions/standards for application messaging through stdin and stdout?
I'm designing and application that spawns another processes to handle some specialized work. The application and the process it spawns will communicate through standard input and standard output (and ...
- 240
4
votes
1
answer
419
views
Which C++ IO interfaces for a complex data source i.e. converter
My question:
When you've got a complex converter like, that takes chunks or large result sets out of a database, converts it into a line by line file/resource in the end, should one either design it ...
- 401
4
votes
1
answer
46k
views
How to let multiple threads write on the same file [closed]
I have got a text file called "vholders.txt".
I am making multiple threads as you can see here ,those threads work with their own given data and at last they write their own output to the vholders....
user196377
29
votes
2
answers
13k
views
Why is universal newlines mode deprecated in Python?
I just noticed that the universal newline feature of file operations seems to be on its way out.
The documentation for Python 3.5 open's mode parameter indicates that it's deprecated:
'U' ...
- 5,578
4
votes
2
answers
598
views
Should a Closeable throw an exception if used after being closed, even if it doesn't have to?
Let's say we have this interface:
public interface OutputChannel extends Closeable {
public void writeOutput(String output);
}
Then there are several classes which implement this interface and ...
- 143
31
votes
6
answers
26k
views
Why do we have to wait for I/O?
It's always been known that Disk operations are slow and we know the reasons why they are slow.
So the question here is why do we have to wait for I/O or why is there such a thing as IOWait, etc.?
I ...
- 571
14
votes
2
answers
1k
views
Readiness vs. Completion Async IO Memory usage?
I was watching this talk about implementing Async IO in Rust and Carl mentions two potential models. Readiness and Completion.
Readiness Model:
you tell the kernel you want to read from a socket
do ...
- 241
3
votes
1
answer
134
views
How "close" should a IO class/logic be to a form or thread that controls the IO device?
I am having a difficult time determining where my IO logic should reside within my application. In this application there are multiple IO devices both USB and serial. I currently have the idea to ...
- 185
2
votes
2
answers
3k
views
Reading using non-blocking IO on a fd asynchronously in C++ (moving from Node.js)
I'm trying to switch my brain from Node.js/Objective-C iOS programming to C++ programming, and it's a little bit taxing. Node.js and Objective-C with iOS do not have a run loop that I am supposed to ...
14
votes
3
answers
22k
views
Reading file during write on linux
As I understand, when a file is being written, the process writing to the file obtains an exclusive lock. So other processes cannot access this file for read.
With the above knowledge, I'm unable ...
- 393
9
votes
1
answer
2k
views
Does it ever make sense to read from and/or write to a bound, listening socket?
I'm experimenting a little with creating a socket server, in PHP. In doing so I'm trying to abstract away the kinds of sockets I think I'll be needing, that I've tentitively named:
ListenSocket — the '...
- 580