If you're running a cottage industry, for efficiency's sake you'll run the first process on each of your raw components first, and store the partially-completed elements in a basket as they're processed. When you've completed that first process, you'll then apply the second process to each element in turn, then the third process, and so on. For small scale production, that's a much MUCH better use of your time and resources that setting up all processes in what's probably a very small area and trying to take components through from end to end.

But if your level of throughput is to be several magnitudes greater, the cottage industry approach doesn't work and you'll want, in a bigger area, to set up a production line. You'll have more overheads, as each stage of the line will need someone to operate it, but you'll save on the need to store large numbers of intermediate components, and you'll save on the time spend putting down and picking up components at each stage too.

Data handling flows can resemble a cottage industry or a production line .... and it depends on how much data you have as to which approach will be the most effective.

In Shell Programming, using a series of operations each of which reads from a file, and saves out to a file with > style redirects, is your cottage industry approach. Using a pipe - | is more production line, with a buffer usually of 4k between the processes.

In Python, functions such as range and readlines return complete lists which you subsequently use - cottage industry. Alternatives such as xrange and xreadlines are generators which are running in parallel with their calling code and so are your production line. And you can write your own generator functions; you can spot them in existing code if you see the yield keyword.

As well as using the cottage industry / production line comparison, I also compare the one-at-a-time approach to being akin to filling up a reservoir from one process, then using the reservoir until it's empty from another. If you have a huge amount of data, then you're likely to overflow your reservoir and have your program fail. However, running the processes at the same time is rather like joining them with a pipe, with a tap that is turned on and off each time a new chunk of data is required. This is how I've successfully trained clients who have data files up to 10Gbytes in size to handle their data in easily written Python scripts.
(written 2006-06-07)

 Y105 - Python - Functions, Modules and Packages
  [96] Variable Scope - (2004-10-22)
  [105] Distance Learning - (2004-10-31)
  [294] Python generator functions, lambdas, and iterators - (2005-04-28)
  [303] Lambdas in Python - (2005-05-06)
  [308] Call by name v call by value - (2005-05-11)
  [340] Code and code maintainance efficiency - (2005-06-08)
  [386] What is a callback? - (2005-07-22)
  [418] Difference between import and from in Python - (2005-08-18)
  [561] Python's Generator functions - (2006-01-11)
  [668] Python - block insets help with documentation - (2006-04-04)
  [745] Python modules. The distribution, The Cheese Shop and the Vaults of Parnassus. - (2006-06-05)
  [775] Do not duplicate your code - (2006-06-23)
  [821] Dynamic functions and names - Python - (2006-08-03)
  [900] Python - function v method - (2006-10-20)
  [912] Recursion in Python - (2006-11-02)
  [913] Python - A list of methods - (2006-11-03)
  [949] Sludge off the mountain, and Python and PHP - (2006-11-27)
  [959] It's the 1st, not the 1nd 1rd or 1th. - (2006-12-01)
  [1134] Function / method parameters with * and ** in Python - (2007-04-04)
  [1163] A better alternative to cutting and pasting code - (2007-04-26)
  [1202] Returning multiple values from a function (Perl, PHP, Python) - (2007-05-24)
  [1464] Python Script - easy examples of lots of basics - (2007-12-08)
  [1784] Global - Tcl, PHP, Python - (2008-09-03)
  [1790] Sharing variables with functions, but keeping them local too - Python - (2008-09-09)
  [1869] Anonymous functions (lambdas) and map in Python - (2008-11-04)
  [1870] What to do with a huge crop of apples - (2008-11-04)
  [1871] Optional and named parameters in Python - (2008-11-05)
  [1879] Dynamic code - Python - (2008-11-11)
  [2011] Conversion of OSI grid references to Eastings and Northings - (2009-01-28)
  [2439] Multiple returns from a function in Python - (2009-10-06)
  [2440] Optional parameters to Python functions - (2009-10-07)
  [2481] Sample code with errors in it on our web site - (2009-10-29)
  [2506] Good example of recursion in Python - analyse an RSS feed - (2009-11-18)
  [2520] Global and Enable - two misused words! - (2009-11-30)
  [2718] Python - access to variables in the outer scope - (2010-04-12)
  [2766] Optional and named parameters to Python functions/methods - (2010-05-15)
  [2878] Program for reliability and efficiency - do not duplicate, but rather share and re-use - (2010-07-19)
  [2929] Passing a variable number of parameters in to a function / method - (2010-08-20)
  [2994] Python - some common questions answered in code examples - (2010-10-10)
  [2998] Using an exception to initialise a static variable in a Python function / method - (2010-10-13)
  [3159] Returning multiple values from a function call in various languages - a comparison - (2011-02-06)
  [3280] Passing parameters to Python functions - the options you have - (2011-05-07)
  [3459] Catching the fishes first? - (2011-09-27)
  [3464] Passing optional and named parameters to python methods - (2011-10-04)
  [3472] Static variables in functions - and better ways using objects - (2011-10-10)
  [3474] Python Packages - groupings of modules. An introduction - (2011-10-11)
  [3662] Finding all the unique lines in a file, using Python or Perl - (2012-03-20)
  [3695] Functions are first class variables in Lua and Python - (2012-04-13)
  [3766] Python timing - when to use a list, and when to use a generator - (2012-06-16)
  [3852] Static variables in Python? - (2012-08-29)
  [3885] Default local - a good choice by the author of Python - (2012-10-08)
  [3931] Optional positional and named parameters in Python - (2012-11-23)
  [3945] vargs in Python - how to call a method with unknown number of parameters - (2012-12-06)
  [4029] Exception, Lambda, Generator, Slice, Dict - examples in one Python program - (2013-03-04)
  [4161] Python varables - checking existance, and call by name or by value? - (2013-08-27)
  [4212] Python functions - an introduction to how they work - (2013-11-16)
  [4361] Multiple yields and no loops in a Python generator? - (2014-12-22)
  [4407] Python - even named code blocks are objects - (2015-01-28)
  [4410] A good example of recursion - a real use in Python - (2015-02-01)
  [4441] Reading command line parameters in Python - (2015-02-23)
  [4448] What is the difference between a function and a method? - (2015-03-04)
  [4645] What are callbacks? Why use them? An example in Python - (2016-02-11)
  [4662] Recursion in Python - the classic example - (2016-03-07)
  [4719] Nesting decorators - (2016-11-02)
  [4722] Embedding more complex code into a named block - (2016-11-04)
  [4724] From and Import in Python - where is the module loaded from? - (2016-11-06)

A167 - Web Application Deployment - Shell Programming (bash)
  [63] Almost like old times - (2004-09-26)
  [64] Shell Script for CGI on the web - (2004-09-26)
  [827] No news is good news with Unix and Linux - (2006-08-10)
  [1287] Work and play at Well House Manor - Football and Shell Shortcuts - (2007-08-02)
  [1345] Perl and Shell coding standards / costs of an IT project - (2007-09-11)
  [1468] Lexical v Arithemetic testing, Bash and Perl - (2007-12-11)
  [1527] Selecting file names in a shell - one word or another - (2008-02-02)
  [1904] Ruby, Perl, Linux, MySQL - some training notes - (2008-11-23)
  [3791] The Kernel, Shells and Daemons. Greek Gods in computing - (2012-07-01)
  [4400] Commenting out an echo killed my bash backup script - (2015-01-19)
  [4487] Starting MySQL. ERROR! The server quit without updating PID file - how we fixed it. - (2015-05-06)
  [4584] Bash ... some new scripts to - handling user input - (2015-11-27)
  [4586] Extending your bash shell with aliases, functions and extra commands - (2015-11-28)
  [4587] shell - bash. Writing conditional tests and statements - the options available - (2015-11-28)

A101 - Web Application Deployment - Linux -An Introduction For Users
  [73] vi - full circle - (2004-10-04)
  [74] pushd and popd - (2004-10-05)
  [152] Aladdin, or careful what you wish. - (2004-12-15)
  [249] An easy way out - (2005-03-17)
  [430] Linux commands - some basics - (2005-08-31)
  [431] File permissions of Linux and Unix systems - (2005-08-31)
  [593] Finding where the disc space has gone - (2006-02-06)
  [659] Web Application Components - (2006-03-28)
  [679] More or less on the edge of the page - (2006-04-11)
  [703] Copying files and preserving ownership - (2006-04-28)
  [710] Linux training Glasgow, Python programming course Dundee - (2006-05-05)
  [711] THE home directory or MY home directory - (2006-05-06)
  [984] Cardinal numbers and magic numbers - (2006-12-14)
  [1012] Moving files between Windows / DOS and Linux / Unix - (2006-12-30)
  [1013] Copy multiple files - confusing error message from cp - (2006-12-30)
  [1068] ls -l report, Linux / Unix - types and permssions - (2007-02-06)
  [1259] Where am I and how did I get here? - (2007-07-05)
  [1288] Linux run states, shell special commands, and directory structures - (2007-08-03)
  [1366] awk - a powerful data extraction and manipulation tool - (2007-09-25)
  [1408] Wireless hotel tips - FTP and Skype connections failing - (2007-10-26)
  [1438] Copy and paste / cut and paste and other vi techniques - (2007-11-20)
  [1651] ls command - favourite options - (2008-05-23)
  [1764] Yank and Push - copy and move in vi - (2008-08-21)
  [1803] FTP passive mode - a sometimes cure for upload hangs - (2008-09-20)
  [1893] Some Linux and Unix tips - (2008-11-18)
  [1897] Keeping on an even keel - (2008-11-21)
  [1902] sstrwxrwxrwx - Unix and Linux file permissions - (2008-11-23)
  [2201] Running straight from the jar, but not from a tar - (2009-05-26)
  [2203] Always use su with minus. And where do programs come from? - (2009-05-27)
  [2299] How much space does my directory take - Linux - (2009-07-20)
  [2300] What does x on a linux directory mean? - (2009-07-21)
  [2479] Accidentally typed ci rather than vi? - (2009-10-27)
  [2494] Making Linux Politically correct - (2009-11-06)
  [2636] Linux - useful tips including history and file name completion - (2010-02-15)
  [2831] Recording (a macro) in vi - (2010-06-27)
  [3179] Oops - I typed ci not vi, and have lost my file ... - (2011-02-21)
  [3256] Displaying a directory or file system tree - Linux - (2011-04-22)
  [3819] Packing a tar, jar or war file - best practise - (2012-07-26)

Back to Getting rid of variables after you have finished with them

Previous and next or Horse's mouth home

Forward to Almost everyone loses

Python 3000 - the next generation
Over zealous police activity?
Want to be a technical trainer in the UK?
Almost everyone loses
Cottage industry or production line data handling methods
Getting rid of variables after you have finished with them
The Fag Packet Design Methodology
Domain Listing Center and Domain Registry of America
We can offer a room, but we can't operate on a dog