Comments on: Fast File Loading http://entland.homelinux.com/blog/2006/10/25/reading-files-as-fas-as-possible/ Code, 3D, Games, Linux and much more... Mon, 09 Nov 2009 03:12:38 +0100 http://wordpress.org/?v=2.8.5 hourly 1 By: Daniel "NeoStrider" Monteiro http://entland.homelinux.com/blog/2006/10/25/reading-files-as-fas-as-possible/comment-page-1/#comment-171521 Daniel "NeoStrider" Monteiro Wed, 24 Jun 2009 01:18:53 +0000 http://entland.homelinux.com/blog/2006/10/25/reading-files-as-fas-as-possible/#comment-171521 I would love to know about anyway to optimize file I/O over Linux. I'm having hard times on linux-powered smartphones, reading files from memory cards. What do you recomend me? I would love to know about anyway to optimize file I/O over Linux.
I’m having hard times on linux-powered smartphones, reading files from memory cards. What do you recomend me?

]]> By: ent http://entland.homelinux.com/blog/2006/10/25/reading-files-as-fas-as-possible/comment-page-1/#comment-168589 ent Mon, 04 May 2009 20:34:12 +0000 http://entland.homelinux.com/blog/2006/10/25/reading-files-as-fas-as-possible/#comment-168589 Big Muscle, did you try the benchmark in your machine? would like to know the results. Big Muscle, did you try the benchmark in your machine? would like to know the results.

]]> By: Big Muscle http://entland.homelinux.com/blog/2006/10/25/reading-files-as-fas-as-possible/comment-page-1/#comment-168569 Big Muscle Sat, 02 May 2009 18:55:14 +0000 http://entland.homelinux.com/blog/2006/10/25/reading-files-as-fas-as-possible/#comment-168569 Hello. Nice article. I just don't understand how did you get similar performance for fread and ReadFile. I'm working on a project where I need to read huge (a few GB) files and fread is about 6x slower than ReadFile, because it does other things as locking critical sections etc. Hello. Nice article. I just don’t understand how did you get similar performance for fread and ReadFile. I’m working on a project where I need to read huge (a few GB) files and fread is about 6x slower than ReadFile, because it does other things as locking critical sections etc.

]]> By: Cyan http://entland.homelinux.com/blog/2006/10/25/reading-files-as-fas-as-possible/comment-page-1/#comment-168548 Cyan Tue, 28 Apr 2009 13:46:11 +0000 http://entland.homelinux.com/blog/2006/10/25/reading-files-as-fas-as-possible/#comment-168548 Hello Yes, I just did it, thanks to your pretty clear explanation. The first thing that striked me was that the timing necessary for completing the asynchronous call was exactly the same as for the synchronous call. After looking a bit further into details, it appears that : - My example focused on writing, rather than reading - I'm expanding the file size as it is written (well, quite common) - Chunks are large (a few MB) - In this case, the OS is single-deciding without notice to treat asynchronous calls as if they were synchronous. So, actually, i cannot use the "space" for additionnal work. I selected to test Writing first because it seems to be where most of the time is spent in my compression program. But maybe i should have a look at it again. Now, with 0.1s spent on a cached 100MB file, i wonder if there is really anything worthwhile to be saved... Hello

Yes, I just did it, thanks to your pretty clear explanation.

The first thing that striked me was that the timing necessary for completing the asynchronous call was exactly the same as for the synchronous call.

After looking a bit further into details, it appears that :
- My example focused on writing, rather than reading
- I’m expanding the file size as it is written (well, quite common)
- Chunks are large (a few MB)
- In this case, the OS is single-deciding without notice to treat asynchronous calls as if they were synchronous.

So, actually, i cannot use the “space” for additionnal work.

I selected to test Writing first because it seems to be where most of the time is spent in my compression program. But maybe i should have a look at it again. Now, with 0.1s spent on a cached 100MB file, i wonder if there is really anything worthwhile to be saved…

]]> By: ent http://entland.homelinux.com/blog/2006/10/25/reading-files-as-fas-as-possible/comment-page-1/#comment-168547 ent Tue, 28 Apr 2009 13:23:16 +0000 http://entland.homelinux.com/blog/2006/10/25/reading-files-as-fas-as-possible/#comment-168547 Yes, more or less the same performance. What you get with the asynchronous methods is space for additional work that can be done in parallel. Any way, you can download the benchmark and try it on your machine. I would appreciate any comments on that. Yes, more or less the same performance. What you get with the asynchronous methods is space for additional work that can be done in parallel.

Any way, you can download the benchmark and try it on your machine. I would appreciate any comments on that.

]]> By: Cyan http://entland.homelinux.com/blog/2006/10/25/reading-files-as-fas-as-possible/comment-page-1/#comment-168545 Cyan Mon, 27 Apr 2009 22:06:04 +0000 http://entland.homelinux.com/blog/2006/10/25/reading-files-as-fas-as-possible/#comment-168545 Strange your article benchmark seems to imply that using Win32 API the "standard" way, with synchronous calls, results in the same performance as the more complex asynchronous method. Is that correct ? W32 3.437 3.467 3.455 +44% Async I/O 3.464 3.475 3.470 +44% Strange
your article benchmark seems to imply that
using Win32 API the “standard” way, with synchronous calls,
results in the same performance as the more complex asynchronous method.

Is that correct ?
W32 3.437 3.467 3.455 +44%
Async I/O 3.464 3.475 3.470 +44%

]]> By: ent http://entland.homelinux.com/blog/2006/10/25/reading-files-as-fas-as-possible/comment-page-1/#comment-167747 ent Mon, 09 Mar 2009 01:14:55 +0000 http://entland.homelinux.com/blog/2006/10/25/reading-files-as-fas-as-possible/#comment-167747 Yes, I was expecting a better efficiency for the memory mapping. But I think in this kind of scenario (reading a big file in linear order) the results are normal. I do not remember better results increasing the window size. But the test was done years ago. The source is provided with the article. May be you want to experiment and share with us the results. I expect to write a revision of the article in a future. Yes, I was expecting a better efficiency for the memory mapping. But I think in this kind of scenario (reading a big file in linear order) the results are normal.

I do not remember better results increasing the window size. But the test was done years ago. The source is provided with the article. May be you want to experiment and share with us the results.

I expect to write a revision of the article in a future.

]]> By: Jon http://entland.homelinux.com/blog/2006/10/25/reading-files-as-fas-as-possible/comment-page-1/#comment-167744 Jon Mon, 09 Mar 2009 00:20:29 +0000 http://entland.homelinux.com/blog/2006/10/25/reading-files-as-fas-as-possible/#comment-167744 I'd expect memory mapping to be the fastest, as that's the only option that will get the file data into the process address space without any copying: the file cache is mapped directly in. Your memory mapping test is adjusting the mapping using 128kb windows - does the speed improve using larger windows? There isn't much reason not to use far larger windows, such as 64mb. I’d expect memory mapping to be the fastest, as that’s the only option that will get the file data into the process address space without any copying: the file cache is mapped directly in.

Your memory mapping test is adjusting the mapping using 128kb windows – does the speed improve using larger windows? There isn’t much reason not to use far larger windows, such as 64mb.

]]> By: Fast File Loading / part II - Load-In-Place | EntBlog http://entland.homelinux.com/blog/2006/10/25/reading-files-as-fas-as-possible/comment-page-1/#comment-166090 Fast File Loading / part II - Load-In-Place | EntBlog Sat, 27 Sep 2008 21:28:20 +0000 http://entland.homelinux.com/blog/2006/10/25/reading-files-as-fas-as-possible/#comment-166090 [...] my first article on Fast File Loading, I described techniques directly related with the hardware and the Operating System to load files [...] [...] my first article on Fast File Loading, I described techniques directly related with the hardware and the Operating System to load files [...]

]]> By: James http://entland.homelinux.com/blog/2006/10/25/reading-files-as-fas-as-possible/comment-page-1/#comment-114678 James Wed, 30 Apr 2008 00:47:51 +0000 http://entland.homelinux.com/blog/2006/10/25/reading-files-as-fas-as-possible/#comment-114678 My above post did not consider the compression case, since it is a different issue. My above post did not consider the compression case, since it is a different issue.

]]>