Home Accessibility Courses Twitter The Mouth Facebook Resources Site Map About Us Contact
 
For 2023 (and 2024 ...) - we are now fully retired from IT training.
We have made many, many friends over 25 years of teaching about Python, Tcl, Perl, PHP, Lua, Java, C and C++ - and MySQL, Linux and Solaris/SunOS too. Our training notes are now very much out of date, but due to upward compatability most of our examples remain operational and even relevant ad you are welcome to make us if them "as seen" and at your own risk.

Lisa and I (Graham) now live in what was our training centre in Melksham - happy to meet with former delegates here - but do check ahead before coming round. We are far from inactive - rather, enjoying the times that we are retired but still healthy enough in mind and body to be active!

I am also active in many other area and still look after a lot of web sites - you can find an index ((here))
Handling huge data files in PHP

I've handled files up to 2 GBytes in size with PHP ... but there are a number of issues to consider.

1. The size of the PHP "footprint" in memory - when you have a huge data file, you cannot simply read it all in with file or file_get_contents (nor with fread, trying to read the file all at once). Rather, you need to iterate through the file in blocks - typically a line at a time (a loop of calls to the fgets function), but I've also worked in 100k blocks.

2. You are very likely to hit the fierce time limit that PHP imposes to stop a program that's looping infinitely from hogging the server for too long. You can solve that one by increasing the time limit. See the set_time_limit function

3. Browsers will also time out (and users of your page get bored too) if you're not able to give a response quite quickly. Options to solve this include sending out a holding page / update periodically (see the manual page on flush for a discussion of this) and - the way I did it - running my PHP analysis of the huge data file as a command line program rather than through the browser.

Looking at the issue more widely, if you do have a huge data file to handle while your users visit your website, it's an excellent idea to preprocess the data to extract all the information they may need as you load the data onto the server / update it OR if that's not going to work for you, to put the data into a MySQL database which is often a much more efficient way of handling huge data that needs regular analysis on the fly.
(written 2006-05-04)

 
Associated topics are indexed as below, or enter http://melksh.am/nnnn for individual articles
H109 - PHP - Input / Output
  [114] Relative or absolute milkman - (2004-11-10)
  [616] printf - a flawed but useful function - (2006-02-22)
  [653] Easy feed! - (2006-03-21)
  [997] Most recent file in a directory - PHP - (2006-12-18)
  [1094] PHP fread - truncated data - (2007-02-27)
  [1096] Sample script - FTP to get a file from within PHP - (2007-03-01)
  [1113] File and URL reading in PHP - (2007-03-20)
  [1442] Reading a file multiple times - file pointers - (2007-11-23)
  [1780] Server overloading - turns out to be feof in PHP - (2008-09-01)
  [2964] An introduction to file handling in programs - buffering, standard in and out, and file handles - (2010-09-21)
  [3029] PHP data sources - other web servers, large data flows, and the client (browser) - (2010-11-04)
  [3159] Returning multiple values from a function call in various languages - a comparison - (2011-02-06)
  [3424] Divide 10000 by 17. Do you get 588.235294117647, 588.24 or 588? - Ruby and PHP - (2011-09-08)
  [4483] Moving from mysql to mysqli - simple worked example - (2015-05-03)


Back to
Linescapes of Mull
Previous and next
or
Horse's mouth home
Forward to
Linux training Glasgow, Python programming course Dundee
Some other Articles
Pictures from my travels
Why reinvent the wheel
THE home directory or MY home directory
Linux training Glasgow, Python programming course Dundee
Handling huge data files in PHP
Linescapes of Mull
A story of goldfish
May day away
Disc Partitioning
Erin Brent - rest in peace
4759 posts, page by page
Link to page ... 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96 at 50 posts per page


This is a page archived from The Horse's Mouth at http://www.wellho.net/horse/ - the diary and writings of Graham Ellis. Every attempt was made to provide current information at the time the page was written, but things do move forward in our business - new software releases, price changes, new techniques. Please check back via our main site for current courses, prices, versions, etc - any mention of a price in "The Horse's Mouth" cannot be taken as an offer to supply at that price.

Link to Ezine home page (for reading).
Link to Blogging home page (to add comments).

You can Add a comment or ranking to this page

© WELL HOUSE CONSULTANTS LTD., 2024: 48 Spa Road • Melksham, Wiltshire • United Kingdom • SN12 7NY
PH: 01144 1225 708225 • EMAIL: info@wellho.net • WEB: http://www.wellho.net • SKYPE: wellho

PAGE: http://www.wellho.info/mouth/709_Hand ... n-PHP.html • PAGE BUILT: Sun Oct 11 16:07:41 2020 • BUILD SYSTEM: JelliaJamb